file caching

hi,

I was redirected here by the novell support newsgroup. even if I'm not 
sure it's the best place to post this message, I didnt't find any 
better. please redirect me if I'm not in the good place.

I'm developping a file-system filter driver to encrypt files 
transparently, and it must work whith NetWare client. What is difficult 
and specific to any file system/network redirector is, when catching a 
read/write request, to know if it will be cached or send over the 
network. Where can I find any information about this technical issue ?

insakda wrote:
> hi,
> 
> I was redirected here by the novell support newsgroup. even if I'm not
> sure it's the best place to post this message, I didnt't find any
> better. please redirect me if I'm not in the good place.
> 
> I'm developping a file-system filter driver to encrypt files
> transparently, and it must work whith NetWare client. What is difficult
> and specific to any file system/network redirector is, when catching a
> read/write request, to know if it will be cached or send over the
> network. Where can I find any information about this technical issue ?

It isn't really the best forum, though it is the appropriate news
server. FWIW, I don't really know where to direct you. There isn't a
system programming forum for the client. FWIW, my understanding is that
the NetWare Client is a filter driver on Win9x and a re-director in Win
NT. If you are only supporting NT based platforms then any filter driver
should see the activity regardless of which IFS driver handles it. You
should be able to confirm this by using the file system monitor
(FileMon) at SysInternals: http://sysinternals.com/ If you load it and
do a bunch of I/O via the client and FileMon shows the activity then a
filter driver should see the activity.

In terms of caching, this should happen below the filter layer. The
NetWare client does cache writes by default and has a read cache. These
are configurable features but you shouldn't depend on them having
particular settings. Nevertheless, I believe you will also see with
FileMon that the write hits a filter driver before it hits the client
and if it is encrypted by the time it reaches the client it will not
matter how long it will be cached in the client.

NB: Take care with this, most encryption schemes are block based and
some are stream based (encryption of subsequent blocks depends on
encryption of previous blocks). Encryption technologies like these are
hard to implement in filters for arbitrary I/O sources since an
application is not required to read or write at a specific location or
with a specific alignment. With these types of encryption an encrypting
filter would have to do the following on a write

Read all blocks containing parts of the write being executed
Decrypt them
Insert the data being written
Encrypt all the blocks
Write the blocks back

Read is similar (doesn't need the re-encrypt or write) and for stream
based encryption technologies you'd have to use super-blocks appropriate
to your implementation.

Note that I'm not a cryptographer and the specific discussion of

David Mair wrote:
> insakda wrote:
> 
>>hi,
>>
>>I was redirected here by the novell support newsgroup. even if I'm not
>>sure it's the best place to post this message, I didnt't find any
>>better. please redirect me if I'm not in the good place.
>>
>>I'm developping a file-system filter driver to encrypt files
>>transparently, and it must work whith NetWare client. What is difficult
>>and specific to any file system/network redirector is, when catching a
>>read/write request, to know if it will be cached or send over the
>>network. Where can I find any information about this technical issue ?
> 
> 
> It isn't really the best forum, though it is the appropriate news
> server. FWIW, I don't really know where to direct you. There isn't a
> system programming forum for the client. FWIW, my understanding is that
> the NetWare Client is a filter driver on Win9x and a re-director in Win
> NT. If you are only supporting NT based platforms then any filter driver
> should see the activity regardless of which IFS driver handles it. You
> should be able to confirm this by using the file system monitor
> (FileMon) at SysInternals: http://sysinternals.com/ If you load it and
> do a bunch of I/O via the client and FileMon shows the activity then a
> filter driver should see the activity.
> 
> In terms of caching, this should happen below the filter layer. The
> NetWare client does cache writes by default and has a read cache. These
> are configurable features but you shouldn't depend on them having
> particular settings. Nevertheless, I believe you will also see with
> FileMon that the write hits a filter driver before it hits the client
> and if it is encrypted by the time it reaches the client it will not
> matter how long it will be cached in the client.
> 
> NB: Take care with this, most encryption schemes are block based and
> some are stream based (encryption of subsequent blocks depends on
> encryption of previous blocks). Encryption technologies like these are
> hard to implement in filters for arbitrary I/O sources since an
> application is not required to read or write at a specific location or
> with a specific alignment. With these types of encryption an encrypting
> filter would have to do the following on a write
> 
> Read all blocks containing parts of the write being executed
> Decrypt them
> Insert the data being written
> Encrypt all the blocks
> Write the blocks back
> 
> Read is similar (doesn't need the re-encrypt or write) and for stream
> based encryption technologies you'd have to use super-blocks appropriate
> to your implementation.
> 
> Note that I'm not a cryptographer and the specific discussion of
> encryption schemes is as I understand it and not offered as fact.

thanks for your relevant response.

you are right, the target is 2K/XP.

My problem is even more complex than this : the cache can not be 
encrypted, because of the direct access to the cache that can be made by 
the applications (file mapping). so when a read/write occur, I have to 
know if it will be cached or not by the file system/network redirector.
And sometimes the NetWare behavior (cache the data or send it directly 
to the server) depends on parameters I can't know : configuration, 
oplock level, etc.

I had the same problem with CIFS and DFS, and I solved it in analysing 
the sources of the smbmrx (given with the ifskit). In my filter driver, 
I have to dereference obscur PVOID pointers. that's not a pretty 
solution but, according to osronline, it's the only one.
I don't have NetWare client sources so I have to find another way to 

insakda wrote:
> David Mair wrote:
> 
>> insakda wrote:
>>
>>> hi,
>>>
>>> I was redirected here by the novell support newsgroup. even if I'm
not
>>> sure it's the best place to post this message, I didnt't find any
>>> better. please redirect me if I'm not in the good place.
>>>
>>> I'm developping a file-system filter driver to encrypt files
>>> transparently, and it must work whith NetWare client. What is
difficult
>>> and specific to any file system/network redirector is, when
catching a
>>> read/write request, to know if it will be cached or send over the
>>> network. Where can I find any information about this technical
issue ?
>>
>>
>>
>> It isn't really the best forum, though it is the appropriate news
>> server. FWIW, I don't really know where to direct you. There isn't a
>> system programming forum for the client. FWIW, my understanding is
that
>> the NetWare Client is a filter driver on Win9x and a re-director in
Win
>> NT. If you are only supporting NT based platforms then any filter
driver
>> should see the activity regardless of which IFS driver handles it. You
>> should be able to confirm this by using the file system monitor
>> (FileMon) at SysInternals: http://sysinternals.com/ If you load it and
>> do a bunch of I/O via the client and FileMon shows the activity then a
>> filter driver should see the activity.
>>
>> In terms of caching, this should happen below the filter layer. The
>> NetWare client does cache writes by default and has a read cache.
These
>> are configurable features but you shouldn't depend on them having
>> particular settings. Nevertheless, I believe you will also see with
>> FileMon that the write hits a filter driver before it hits the client
>> and if it is encrypted by the time it reaches the client it will not
>> matter how long it will be cached in the client.
>>
>> NB: Take care with this, most encryption schemes are block based and
>> some are stream based (encryption of subsequent blocks depends on
>> encryption of previous blocks). Encryption technologies like these are
>> hard to implement in filters for arbitrary I/O sources since an
>> application is not required to read or write at a specific location or
>> with a specific alignment. With these types of encryption an
encrypting
>> filter would have to do the following on a write
>>
>> Read all blocks containing parts of the write being executed
>> Decrypt them
>> Insert the data being written
>> Encrypt all the blocks
>> Write the blocks back
>>
>> Read is similar (doesn't need the re-encrypt or write) and for stream
>> based encryption technologies you'd have to use super-blocks
appropriate
>> to your implementation.
>>
>> Note that I'm not a cryptographer and the specific discussion of
>> encryption schemes is as I understand it and not offered as fact.
> 
> 
> thanks for your relevant response.
> 
> you are right, the target is 2K/XP.
> 
> My problem is even more complex than this : the cache can not be
> encrypted, because of the direct access to the cache that can be made by
> the applications (file mapping). so when a read/write occur, I have to
> know if it will be cached or not by the file system/network redirector.
> And sometimes the NetWare behavior (cache the data or send it directly
> to the server) depends on parameters I can't know : configuration,
> oplock level, etc.

I would test this with a filter driver. I doubt that a memory mapped
file gives direct read access to the client's cache. That cache isn't
really much more than a way of collecting data so that grouped I/O
operations can be executed and improve network efficiency. By the time
something is in there then it should be in structure unknown to the OS
so unless read/write of a mapped file is implemented by a IFS driver
(which seems reasonably likely though). However, a memory mapped file
must surely perform actual reads into actual OS cache buffers and those
OS cache buffers are the ones that appear in the virtual address space
of the application. Otherwise it would seem to be impossible for the OS
to provide any co-ordinated memory mapped file features - lots of
drivers would be implementing specific Win32 functions. Load several IFS
drivers and you have several implementations of the same Win32 function.
I really would test to see if this problem actually exists for the NCP
client's caches. Turn on all the caching options and see if you can
encrypt memory mapped and non-memory mapped files. I suppose you can't
make an exlusive assertion that there are no problems but I think it
will work because I don't think those cache buffers are going to be the
ones that the OS uses for memory mapped or non-memory mapped file
buffers. Once they pass through your filter I believe they are likely to
be in unknown state for the OS file I/O functions.

Do I understand that you are trying only to encrypt the physical I/O
buffers so that only the persistent storage is encrypted? If that is the
case but you can live with unencrypted data going over the wire then
will the encrypted file system support in NSS suffice to resolve your
problem. If you have to have encryption on the wire can you use an
encrypted network path. Those would be worth considering only if you are
trying to deploy a specific solution but obviously would be of little
use in creating a product that creates these features.

> I had the same problem with CIFS and DFS, and I solved it in analysing
> the sources of the smbmrx (given with the ifskit). In my filter driver,
> I have to dereference obscur PVOID pointers. that's not a pretty
> solution but, according to osronline, it's the only one.
> I don't have NetWare client sources so I have to find another way to
> solve this problem :)

I doubt I could get you the client source but, depending on your
project, it is is feasible (note that this is neither an offer or a
statement on behalf of Novell). There have been some third party
products that Novell supported by providing proprietary source under
NDA. Whether or not that is even likely to be considered in your case is
beyond my ability to comment and you'd have to approach Novell via the