Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Experimental - Replicated chunk storage #6881

Draft
wants to merge 9 commits into
base: master
Choose a base branch
from

Conversation

sachin-j-joshi
Copy link
Contributor

Experimental -
Change log description

  1. Implement ReplicatedChunkStorage that wraps multiple instances of chunk storage acting as an ensemble.
    Write quorum - all replicas must succeed.
    Read quorum - at least one replica succeeds.
  2. Hack ECS ChunkStorage factory to allow specifying multiple end points and create ensemble
  3. Add end to end integration test for ECS
  4. Refactor test code to pull out FlakyChunkStorage so that it cane be used widely in other tests.

Purpose of the change
(e.g., Fixes #666, Closes #1234)

What the code does
(Detailed description of the code changes)

How to verify it
(Optional: steps to verify that the changes are effective)

Signed-off-by: Sachin Joshi <sachin.joshi@emc.com>
Signed-off-by: Sachin Joshi <sachin.joshi@emc.com>
@codecov
Copy link

codecov bot commented Sep 22, 2022

Codecov Report

Base: 86.34% // Head: 86.31% // Decreases project coverage by -0.03% ⚠️

Coverage data is based on head (209cc1b) compared to base (8a66db8).
Patch coverage: 65.25% of modified lines in pull request are covered.

Additional details and impacted files
@@             Coverage Diff              @@
##             master    #6881      +/-   ##
============================================
- Coverage     86.34%   86.31%   -0.04%     
- Complexity    15910    15947      +37     
============================================
  Files          1027     1029       +2     
  Lines         59273    59386     +113     
  Branches       5997     6007      +10     
============================================
+ Hits          51180    51257      +77     
- Misses         4955     4984      +29     
- Partials       3138     3145       +7     
Impacted Files Coverage Δ
...ega/storage/extendeds3/ExtendedS3ChunkStorage.java 83.68% <0.00%> (-1.21%) ⬇️
...ain/java/io/pravega/storage/s3/S3ChunkStorage.java 86.98% <0.00%> (-1.05%) ⬇️
...tore/storage/chunklayer/AsyncBaseChunkStorage.java 87.04% <ø> (ø)
...e/chunklayer/ChunkStorageUnavailableException.java 0.00% <0.00%> (ø)
...age/extendeds3/ExtendedS3SimpleStorageFactory.java 57.14% <14.28%> (-42.86%) ⬇️
...ga/storage/extendeds3/ExtendedS3StorageConfig.java 52.17% <34.37%> (-42.28%) ⬇️
...ore/storage/chunklayer/ReplicatedChunkStorage.java 91.54% <91.54%> (ø)
...hared/security/crypto/StrongPasswordProcessor.java 87.50% <0.00%> (-6.25%) ⬇️
...ver/containers/StreamSegmentContainerMetadata.java 96.81% <0.00%> (-0.64%) ⬇️
...a/segmentstore/server/logs/OperationProcessor.java 91.42% <0.00%> (-0.36%) ⬇️
... and 7 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

Signed-off-by: Sachin Joshi <sachin.joshi@emc.com>
Signed-off-by: Sachin Joshi <sachin.joshi@emc.com>
…icated_chunk_storage

Signed-off-by: Sachin Joshi <sachin.joshi@emc.com>

# Conflicts:
#	segmentstore/storage/src/test/java/io/pravega/segmentstore/storage/chunklayer/FlakinessPredicate.java
#	segmentstore/storage/src/test/java/io/pravega/segmentstore/storage/chunklayer/FlakyChunkStorage.java
#	segmentstore/storage/src/test/java/io/pravega/segmentstore/storage/chunklayer/FlakyInterceptor.java
#	segmentstore/storage/src/test/java/io/pravega/segmentstore/storage/chunklayer/FlakySnapshotInfoStore.java
#	segmentstore/storage/src/test/java/io/pravega/segmentstore/storage/chunklayer/SystemJournalOperationsTests.java
…eException.

Signed-off-by: Sachin Joshi <sachin.joshi@emc.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add unit tests for RocksDBCache
1 participant