FastRsyncNet - C# delta syncing library

The Fast Rsync .NET library is Rsync implementation derived from Octodiff tool.

Unlike the Octodiff which is based on SHA1 algorithm, the FastRsyncNet uses xxHash64 as a default algorithm. Usage of xxHash64 allows for significant faster calculations and smaller signature size.

FastRsyncNet supports also SHA1 and is 100% compatible with signatures and deltas produced by Octodiff.

Since version 2.0.0 the signature and delta format has changed. FastRsyncNet 2.x is still able to work with signatures and deltas from FastRsync 1.x and Octodiff. However, files made with FastRsyncNet 2.x are not going to be recognized by FastRsyncNet 1.x.

Install

Add To project via NuGet:

1. Right click on a project and click 'Manage NuGet Packages'.
2. Search for 'FastRsyncNet' and click 'Install'.

Examples

Calculating signature

using FastRsync.Signature;

...

var signatureBuilder = new SignatureBuilder();
using (var basisStream = new FileStream(basisFilePath, FileMode.Open, FileAccess.Read, FileShare.Read))
using (var signatureStream = new FileStream(signatureFilePath, FileMode.Create, FileAccess.Write, FileShare.Read))
{
    signatureBuilder.Build(basisStream, new SignatureWriter(signatureStream));
}

Calculating delta

using FastRsync.Delta;

...

var delta = new DeltaBuilder();
builder.ProgressReport = new ConsoleProgressReporter();
using (var newFileStream = new FileStream(newFilePath, FileMode.Open, FileAccess.Read, FileShare.Read))
using (var signatureStream = new FileStream(signatureFilePath, FileMode.Open, FileAccess.Read, FileShare.Read))
using (var deltaStream = new FileStream(deltaFilePath, FileMode.Create, FileAccess.Write, FileShare.Read))
{
    delta.BuildDelta(newFileStream, new SignatureReader(signatureStream, delta.ProgressReporter), new AggregateCopyOperationsDecorator(new BinaryDeltaWriter(deltaStream)));
}

Patching (applying delta)

using FastRsync.Delta;

...

var delta = new DeltaApplier
        {
            SkipHashCheck = true
        };
using (var basisStream = new FileStream(basisFilePath, FileMode.Open, FileAccess.Read, FileShare.Read))
using (var deltaStream = new FileStream(deltaFilePath, FileMode.Open, FileAccess.Read, FileShare.Read))
using (var newFileStream = new FileStream(newFilePath, FileMode.Create, FileAccess.ReadWrite, FileShare.Read))
{
    delta.Apply(basisStream, new BinaryDeltaReader(deltaStream, progressReporter), newFileStream);
}

Calculating signature on Azure blobs

FastRsyncNet might not work on Azure Storage emulator due to issues with stream seeking.

using FastRsync.Signature;

...

var storageAccount = CloudStorageAccount.Parse("azure storage connectionstring");
var blobClient = storageAccount.CreateCloudBlobClient();
var blobsContainer = blobClient.GetContainerReference("containerName");
var basisBlob = blobsContainer.GetBlockBlobReference("blobName");

var signatureBlob = container.GetBlockBlobReference("blob_signature");

var signatureBuilder = new SignatureBuilder();
using (var signatureStream = await signatureBlob.OpenWriteAsync())
using (var basisStream = await basisBlob.OpenReadAsync())
{
    await signatureBuilder.BuildAsync(basisStream, new SignatureWriter(signatureStream));
}

GZip compression that is rsync compatible

If you synchronize compressed file, a small change in a compressed file may force rsync algorithm to synchronize whole compressed file, instead of just the changed blocks. To fix this, a custom GZip compression method may be used that periodically reset the compressor state to make it block-sync friendly. Install FastRsyncNet.Compression package and use following method:

FastRsync.Compression.GZip.Compress(Stream sourceStream, Stream destStream)

To uncompress you may use any GZip method (e.g. System.IO.Compression.GZipStream).