View Issue Details

IDProjectCategoryView StatusLast Update
0000495RsyncBugpublic2017-11-29 19:41
ReporterLewisR Assigned ToSteven Levine  
PrioritynormalSeverityminorReproducibilityalways
Status feedbackResolutionopen 
Product Version3.0.6 
Summary0000495: Rsync has trouble with --xattrs when whitespace is present in path and/or no EAs present (certain files)
DescriptionThis is with Steve's 3.0.8 build of 2011-04-19.

(Originally, we had --no-xattrs specified due to another issue with EAs which I can't recall offhand.)

With --xattrs specified, and the rsync pass hits a directory with a space, I'm seeing a DosQueryPathInfo error 8 at lib/sysxattrs.c(241). This error does not occur on *all* directories with whitespaces, nor on all files without EAs, so I'm a bit stymied as to what the exact cause may be (example file attached, through the miracle of mdirs).
Additional InformationCommand is run from cron:

c:\opt\rsync\bin\rsync.exe -r --xattrs --times --stats --human-readable --log-file=c:/var/log/rsync_cgp.log --exclude-from=c:/MPTN/ETC/rsync-cgp-exclude.lst j:/Communigate j:/CGP_Backup >nul 2>&1

Log shows:

2011/11/06 00:32:52 [4654] [generator] unigetxattr: DosQueryPathInfo(Communigate/Accounts/lgrosenthal.macnt/OS2 Wireless.folder/all-messages.mdir/102097-S_______-20031123115013-89, FIL_QUERYEASFROMLIST) failed with error 8 at lib/sysxattrs.c(241)

Dir (from 4OS2 3.0.7) shows:

 Volume in drive J is DATA Serial number is 657E:D5BA
 Directory of J:\Communigate\Accounts\lgrosenthal.macnt\OS2 Wireless.folder\all-messages.mdir\102097-S_______-20031123115013-89

 8-01-09 23:44 3,538 0 102097-S_______-20031123115013-89
           3,538 bytes in 1 file and 0 dirs
           4,096 bytes allocated

(file attached)

(Steve, you may note that this is one of the messages which we massaged with your Perl script when I migrated to Communigate Pro; in fact, all of these older files are generating this error. These files were originally contiguous, i.e., contained in a single mbox, and later, split into separate files in an mdir via another Perl script - along with other mbox files.)
TagsNo tags attached.
Attached Files
102097-S_______-20031123115013-89 (3,538 bytes)   
X-UIDL: 2342
X-Mozilla-Keys:                                                                                 
Return-Path: os2-wireless_users-owner@2rosenthals.com
Received: from mail.2rosenthals.com (localhost [127.0.0.1] ) by mail.2rosenthals.com
    (Hethmon Brothers Smtpd) ; Sun, 23 Nov 2003 11:50:13 -0500
Received: from mail1.no-ip.com (mail1.no-ip.com [63.215.241.221] ) by mail.2rosenthals.com
    (Hethmon Brothers Smtpd) ; Sun, 23 Nov 2003 11:50:12 -0500
X-Envelope-To: <os2-wireless_users@2rosenthals.com>
Received: (qmail 10904 invoked by uid 89); 23 Nov 2003 16:49:49 -0000
Received: from unknown (HELO MAIL03.toast.net) (206.244.185.10)
  by mail1.no-ip.com with SMTP; 23 Nov 2003 16:49:49 -0000
Received: from 2rosenthals.com (unverified [206.149.148.84]) by MAIL03.toast.net
 (Vircom SMTPRS 3.0.273) with ESMTP id <B0057899567@MAIL03.toast.net> for <os2-wireless_users@2rosenthals.com>;
 Sun, 23 Nov 2003 11:48:39 -0500
Message-ID: <3FC0E53F.9020703@2rosenthals.com>
Organization: Rosenthal & Rosenthal
User-Agent: Mozilla/5.0 (OS/2; U; Warp 4.5; en-US; rv:1.4.1) Gecko/20030909
X-Accept-Language: en-us, en
MIME-Version: 1.0
References: <auto-000208779654@admin.nni.com>
In-Reply-To: <auto-000208779654@admin.nni.com>
Content-Type: text/plain; charset=us-ascii; format=flowed
Content-Transfer-Encoding: 7bit
Date:  Sun, 23 Nov 2003 11:50:07 -0500
Sender: os2-wireless_users-owner <os2-wireless_users-owner@2rosenthals.com>
X-Listname: os2-wireless_users@2rosenthals.com
Reply-To: os2-wireless_users@2rosenthals.com
From:  Lewis G Rosenthal <os2-wireless_users@2rosenthals.com>
To:  os2-wireless_users@2rosenthals.com
Subject: [OS2Wireless] OS/2 software's site survey function
X-List-Unsubscribe: Send email to mailusers-request@2rosenthals.com
X-List-Owner: mailusers-owner@2rosenthals.com

Stan, it's been my experience recently, that when WEP is enabled on the 
AP and the client has WEP turned off, that I haven't even been able to 
associate with it. I suppose it could be a driver level thing; I've only 
tried this with the IBM High Rate and the Cisco 350 cards. This was also 
under Windows XP, BTW.

On 11/22/2003 08:51 am, Stanley Sidlov thus wrote :

>On Fri, 21 Nov 2003 20:25:19 -0500, Lewis G Rosenthal wrote:
>
>  
>
>>  4. If the AP has WEP enabled, even if it is beaconing, you won;t be
>>     able to associate with it (or see its SSID in WiFiState).
>>    
>>
>
> why do I remember attaching and having frame errors when the wep is incorrect? In windows, that is?
>
>
>Stan Sidlov
>--
>BOD, Warpstock, Inc.
>
>Did YOU go to WARPSTOCK 2003 ?  No?! You missed a lot!  Make a bid for 2004! 
>
> http://www.warpstock.org
>  
>
-- 
Lewis 
------------------------------------------------------------ 
Lewis G Rosenthal, CNA 
   Rosenthal & Rosenthal 
Accountants / Network Consultants 
  New York / Northern Virginia           www.2rosenthals.com
Team OS/2  / NetWare Users International      www.novell.com
------------------------------------------------------------ 
This OS/2 system (Apollo) uptime is 0 days 00:17 hours and 20 seconds


=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

To unsubscribe from this list, send a message to
steward@2rosenthals.com with the command
"unsubscribe os2-wireless_users" in the body
(omit the quotes).

For help with other commands, send a message
to steward@2rosenthals.com with the command
"help" in the body (omit the quotes).

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=



Activities

Steven Levine

2011-12-31 00:16

manager   ~0002089

Since the pathname in the the error message is relative, it implies that the problem is with EAs on the destination file. What does the dir command have to say out this file?

LewisR

2011-12-31 03:37

developer   ~0002091

Volume in drive J is DATA Serial number is 657E:D5BA
 Directory of J:\CGP_Backup\Communigate\Accounts\lgrosenthal.macnt\OS2 Wireless
.folder\all-messages.mdir\102097-S_______-20031123115013-89

 8-01-09 23:44 3,538 0 102097-S_______-20031123115013-89
           3,538 bytes in 1 file and 0 dirs
           4,096 bytes allocated
  16,920,453,120 bytes (15GB) free

No EAs, as expected. Only the archive bit is set on the backup file, as well. Do you think this could be some underlying filesystem corruption? The volume had already undergone that long chkdsk pass which we discussed on one of our mutual lists (that was 10/21, IIRC).

The thing is that this file was already copied, and had no EAs on eithe rthe original or the first copy. It hasn't changed since then, so rsync should have just skipped over it (unless it had some difficulty accessing the file for comparison).

The only things in the exclude list are the queues & logs directories, too, BTW. Oh, and no EAs on the directory itself (source or destination), either.

Steven Levine

2011-12-31 05:35

manager   ~0002098

Let's see if

 http://mantis.smedley.info/file_download.php?file_id=221&type=bug

can tell us anything. It adds the failing EA name to the error report.

LewisR

2011-12-31 21:27

developer   ~0002118

Hehehe...

No details, and no error (at least, not this error) running rsync-3.0.8-20111230-shl. Let's close this as resolved, unless you think that whatever changed between 20110419 and 20111231 regarding xattrs could not have addressed this.

To test, after the first rsync pass didn;t copy any files (Doh! because they were already there!), I deleted about 50 of the backup copies (including the sample, referenced in this bug) in the problem directory, and waited for the next hourly pass. The entire group of files copied without error.

Thanks!

Steven Levine

2011-12-31 21:38

manager   ~0002119

I recommend we leave this ticket open for a bit. I did not change anything that could have resolved the error you reported.

LewisR

2012-01-09 19:24

developer   ~0002158

Yep... Hit it again:

2012/01/09 13:14:29 [8556] [sender] unigetxattr: DosQueryPathInfo(CGP_Backup/Communigate/Domains/driven-zen.org/bulkimages.macnt/INBOX.mdir/42350-________-20100509231812-372, FIL_QUERYEASFROMLIST) for FLAGS failed with error 8 at lib/sysxattrs.c(241)

This time, no whitespaces in the path, and this file should not have been in use by any other process, as it's:

1. An old message file (2010);
2. Already in the backup directory (CGP_Backup);
3. Unchanged since originally backed up, and the hourly backup pass would not have had this open at the same time.

A dir on the file shows:

 Volume in drive J is DATA Serial number is 657E:D5BA
 Directory of J:\CGP_Backup\Communigate\Domains\driven-zen.org\bulkimages.macnt
\INBOX.mdir\42350-________-20100509231812-372

 5-09-10 19:18 12,387 124 42350-________-20100509231812-372
          12,387 bytes in 1 file and 0 dirs

The 124 bytes of EAs appear to all be LIBC EAs.

rsync-3.0.8-20111230-shl made two attempts (it would appear from the log) to access this file and two others in that directory, all of which failed with the same error. These files were not copied to the destination at all.

rsync command used:

c:\opt\rsync\bin\rsync.exe --archive --xattrs --os2-perms --del --itemize-changes --times --stats --human-readable --log-file=c:/var/log/rsync_backup_j.log j:/ e:/

I don't know how the LIBC EAs would have even been attached to the original file, as really none of the CGP mail files should have any EAs on them, however, it appears that every file touched by rsync now has LIBC EAs attached. In fact, looking more closely, I find that I actually have duplicate files, one version without EAs and another with EAs (the ones with EAs do *not* exist in the live CGP mail directory). Here's an example (from a different mailbox backup):

10-05-11 2:58 75,268 0 1-S_______-20111005065823-1262
10-05-11 2:58 75,268 124 1-________-20111005065823-1262
10-05-11 3:21 32,351 0 2-S_______-20111005072112-776
10-05-11 3:21 32,351 124 2-________-20111005072112-776
10-05-11 3:44 3,839 0 3-S_______-20111005074412-104
10-05-11 3:44 3,839 124 3-________-20111005074412-104
10-05-11 4:33 13,396 0 4-S_______-20111005083312-322
10-05-11 4:33 13,396 124 4-________-20111005083312-322
10-05-11 9:00 16,640 0 5-S_______-20111005130031-639
10-05-11 9:00 16,640 124 5-________-20111005130031-639
10-05-11 9:06 3,821 0 6-S_______-20111005130636-109
10-05-11 9:06 3,821 124 6-________-20111005130636-109
10-05-11 9:39 47,094 0 7-S_______-20111005133952-785
10-05-11 9:39 47,094 124 7-________-20111005133952-785
10-05-11 10:24 15,184 0 8-S_______-20111005142455-555
10-05-11 10:24 15,184 124 8-________-20111005142455-555
10-05-11 10:28 2,587 0 9-S_______-20111005142811-62
10-05-11 10:45 649 0 10-S_______-20111005144514-15

and the live mailbox:

10-05-11 2:58 75,268 0 1-S_______-20111005065823-1262
10-05-11 3:21 32,351 0 2-S_______-20111005072112-776
10-05-11 3:44 3,839 0 3-S_______-20111005074412-104
10-05-11 4:33 13,396 0 4-S_______-20111005083312-322
10-05-11 9:00 16,640 0 5-S_______-20111005130031-639
10-05-11 9:06 3,821 0 6-S_______-20111005130636-109
10-05-11 9:39 47,094 0 7-S_______-20111005133952-785
10-05-11 10:24 15,184 0 8-S_______-20111005142455-555
10-05-11 10:28 2,587 0 9-S_______-20111005142811-62
10-05-11 10:45 649 0 10-S_______-20111005144514-15
10-05-11 10:45 5,539 0 11-S_______-20111005144552-141
10-05-11 11:26 647 0 12-S_______-20111005152616-15
10-05-11 11:58 649 0 13-S_______-20111005155826-15
10-05-11 12:10 131,806 0 14-S_______-20111005161034-1789
10-05-11 12:22 16,963 0 15-S_______-20111005162212-399
10-05-11 13:09 21,189 0 16-S_______-20111005170920-490
10-05-11 13:25 10,021 0 17-S_______-20111005172505-331
10-05-11 13:41 3,335 0 18-S_______-20111005174144-98

Not sure why the "S" was replaced with a "_" when the EAs were tacked on. I'm happy to open a separate bug for the duplicate files or perhaps we should create a meta-bug for EA-related issues?

LewisR

2012-01-09 19:33

developer   ~0002159

I understand why I have duplicate files. The filenames changed as a result of the state of the message changing (not read -> read). Now to figure out how to get rsync to replace the old versions with the ones of changed state, or better, skip them entirely. Still, we should not be *adding* EAs to the copied files when --xattrs is specified, should we?

Steven Levine

2012-01-10 08:36

manager   ~0002162

I am not sure what you mean by "we should not be *adding* EAs to the copied files when --xattrs is specified." Rsync sync's EAs. It does not add EAs that do not exist at the source. If something "added" EAs to the source file, it is unlikely that it was rsync.

Note that there may be an errant libc based app out there that does add EAs to every file it accesses. I have had one report of a user system that had libc EAs attached to every file on the boot volume.

LewisR

2012-01-10 16:54

developer   ~0002163

CGP is not a gcc app. None of CGP's files have *any* EAs on them, yet when running rsync.exe --xattrs, *all* backed up copies get libc EAs (exactly 124 bytes) attached.

This, though, as I said in comment #2159, is not the real issue insofar as multiple copies of the same content are concerned. It *is*, however, an issue either for libc or rsync, apparently.

What I've found is that while I haven't been able to get 3.0.8 to tack on the EAs (perhaps those came from a previous build?) it *will* consistently do it when creating new directories. I'll open a separate issue for that.

Steven Levine

2012-01-10 21:33

manager   ~0002164

I finally figured out what you did to generate the EAs. Way back when you were running rsync without --xattrs. This means that libc will add the libc private EAs and rsync will leave them alone. IMO, this is appropriate. It's not different that if rsync was run without --perms. The destination file permissions will be set to to runtime defaults.

Once you started using --xattrs, rsync brought the EAs into sync for existing files. Files that no longer existed at the source remaining unchanged. If you want them to go away you can use my StripLibcEAs script or the equivalent.

Note that libc064 supports the LIBC_UNIX_EAS environment variable which allows the libc EAs logic to be suppressed for specific volumes. Unfortunately, libc064 is not yet sufficiently stable for rsync use, so we can not yet use this.

The directory EAs are a known issue. I won't bore you with the gory details, but currently it takes two rsync runs to get the directory EAs into sync. Once we have a working libc with LIBC_UNIX_EAS support, I might be able to use this to avoid the spurious EAs.

LewisR

2012-01-11 02:55

developer   ~0002171

Okay. Understood. So, we're still back where we started in this bug, trying to figure out why we get the error 8 on some files and not others. Per my PM, as long as we're not getting EA data *replaced*, the fact that we're adding 124 extra bytes of EAs is of little consequence. The presence (or lack thereof) of these EAs is not what is causing the multiple copies of the CGP files, as we've determined, which was my initial concern.

Thanks for the clarification.

asavage

2013-03-12 17:51

reporter   ~0002416

Last edited: 2013-03-12 18:12

Where can I learn more about LIBC_UNIX_EAS ?

Has libc065.dll support for the env var LIBC_UNIX_EAS

I see at:
http://svn.netlabs.org/libc/ticket/43

"the value of the variable is on the forms:

   set LIBC_UNIX_EAS=!c,!e-f,d,g-h
"

Does this work now?

Oh, and where may I find the StripLibcEAs script?

Steven Levine

2013-03-12 20:38

manager   ~0002417

There's not a lot of external documentation.

  http://svn.netlabs.org/libc/search?q=LIBC_UNIX_EAS

will tell you a bit more than you already know. As the query shows, the last change to the support was http://svn.netlabs.org/libc/changeset/3697.

LIBC065 was the first release to formally support the feature.

I'll send you a copy of StripLibcEAs privately with some comments. It's not 100% generic.

Steven Levine

2017-11-29 19:41

manager   ~0003153

Please retest with

 http://www.warpcave.com//betas/rsync-3.0.9.1-20171111-shl.zip

Issue History

Date Modified Username Field Change
2011-11-06 05:58 LewisR New Issue
2011-11-06 05:58 LewisR File Added: 102097-S_______-20031123115013-89
2011-12-31 00:16 Steven Levine Note Added: 0002089
2011-12-31 00:22 Steven Levine Assigned To => Steven Levine
2011-12-31 00:22 Steven Levine Status new => assigned
2011-12-31 03:37 LewisR Note Added: 0002091
2011-12-31 05:35 Steven Levine Note Added: 0002098
2011-12-31 05:35 Steven Levine Status assigned => feedback
2011-12-31 21:27 LewisR Note Added: 0002118
2011-12-31 21:27 LewisR Status feedback => assigned
2011-12-31 21:38 Steven Levine Note Added: 0002119
2012-01-09 19:24 LewisR Note Added: 0002158
2012-01-09 19:33 LewisR Note Added: 0002159
2012-01-10 08:36 Steven Levine Note Added: 0002162
2012-01-10 16:54 LewisR Note Added: 0002163
2012-01-10 21:33 Steven Levine Note Added: 0002164
2012-01-11 02:55 LewisR Note Added: 0002171
2013-03-12 17:51 asavage Note Added: 0002416
2013-03-12 18:12 asavage Note Edited: 0002416
2013-03-12 20:38 Steven Levine Note Added: 0002417
2017-11-29 19:41 Steven Levine Status assigned => feedback
2017-11-29 19:41 Steven Levine Note Added: 0003153