Multi-database client implementation by vladvildanov · Pull Request #3784 · redis/redis-py (original) (raw)

Pull Request check-list

Please make sure to review and check all of these items:

Do tests and lints pass with this change?
Do the CI tests pass with this change (enable it first in your forked repo and wait for the github action build to finish)?
Is the new or changed code fully tested?
Is a documentation update included (if this change modifies existing APIs, or introduces new ones)?
Is there an example added to the examples folder (if applicable)?

NOTE: these things are not required to open a PR and can be done afterwards / while the PR is open.

Description of change

Following feature provides a client to communicate with multiple eventually consistent database, whereas one active and N passive databases. Client is able to failover and fallback between databases in seamless way and provides a variety of configuration options depends on client application preferences

Added Database, Healthcheck, CircuitBreaker, FailureDetector
Added DatabaseSelector, exceptions, refactored existing entities
Added MultiDbConfig
Added DatabaseConfig
Added DatabaseConfig test coverage
Renamed DatabaseSelector into FailoverStrategy
Added CommandExecutor
Updated healthcheck to close circuit on success
Added thread-safeness
Added missing thread-safeness
Added missing thread-safenes for dispatcher
Refactored client to keep databases in WeightedList
Added database CRUD operations
Added on-fly configuration
Added background health checks
Added background healthcheck + half-open event
Refactored background scheduling
Refactored healthchecks
Removed code repetitions, fixed weight assignment, added loops enhancement, fixed data structure
Refactored configuration
Refactored failure detector
Refactored retry logic
Added scenario tests
Added pybreaker optional dependency
Added pybreaker to dev dependencies
Rename tests directory
Remove redundant checks
Handle retries if default is not set
Removed all Sentinel related

Added Database, Healthcheck, CircuitBreaker, FailureDetector
Added DatabaseSelector, exceptions, refactored existing entities
Added MultiDbConfig
Added DatabaseConfig
Added DatabaseConfig test coverage
Renamed DatabaseSelector into FailoverStrategy
Added CommandExecutor
Updated healthcheck to close circuit on success
Added thread-safeness
Added missing thread-safeness
Added missing thread-safenes for dispatcher
Refactored client to keep databases in WeightedList
Added database CRUD operations
Added on-fly configuration
Added background health checks
Added background healthcheck + half-open event
Refactored background scheduling
Added support for Active-Active pipeline
Refactored healthchecks
Added Pipeline testing
Added support for transactions
Removed code repetitions, fixed weight assignment, added loops enhancement, fixed data structure
Added missing doc blocks
Refactored configuration
Refactored failure detector
Refactored retry logic
Added scenario tests
Added pybreaker optional dependency
Added pybreaker to dev dependencies
Rename tests directory
Added scenario tests for Pipeline and Transaction
Added handling of ConnectionRefusedError, added timeouts so cluster could recover
Increased timeouts
Refactored integration tests
Fixed property name
Removed sentinels
Removed unused method

Added Database, Healthcheck, CircuitBreaker, FailureDetector
Added DatabaseSelector, exceptions, refactored existing entities
Added MultiDbConfig
Added DatabaseConfig
Added DatabaseConfig test coverage
Renamed DatabaseSelector into FailoverStrategy
Added CommandExecutor
Updated healthcheck to close circuit on success
Added thread-safeness
Added missing thread-safeness
Added missing thread-safenes for dispatcher
Refactored client to keep databases in WeightedList
Added database CRUD operations
Added on-fly configuration
Added background health checks
Added background healthcheck + half-open event
Refactored background scheduling
Added support for Active-Active pipeline
Refactored healthchecks
Added Pipeline testing
Added support for transactions
Removed code repetitions, fixed weight assignment, added loops enhancement, fixed data structure
Added missing doc blocks
Added support for Pub/Sub in MultiDBClient
Refactored configuration
Refactored failure detector
Refactored retry logic
Added scenario tests
Added pybreaker optional dependency
Added pybreaker to dev dependencies
Rename tests directory
Added scenario tests for Pipeline and Transaction
Added handling of ConnectionRefusedError, added timeouts so cluster could recover
Increased timeouts
Refactored integration tests
Added scenario tests for Pub/Sub
Updated healthcheck retry
Increased timeout to avoid unprepared state before tests
Added backoff retry and changed timeouts
Added retry for healthchecks to avoid fluctuations
Changed retry configuration for healthchecks
Fixed property name
Added check for thread results

Added Database, Healthcheck, CircuitBreaker, FailureDetector
Added DatabaseSelector, exceptions, refactored existing entities
Added MultiDbConfig
Added DatabaseConfig
Added DatabaseConfig test coverage
Renamed DatabaseSelector into FailoverStrategy
Added CommandExecutor
Updated healthcheck to close circuit on success
Added thread-safeness
Added missing thread-safeness
Added missing thread-safenes for dispatcher
Refactored client to keep databases in WeightedList
Added database CRUD operations
Added on-fly configuration
Added background health checks
Added background healthcheck + half-open event
Refactored background scheduling
Added support for Active-Active pipeline
Refactored healthchecks
Added Pipeline testing
Added support for transactions
Removed code repetitions, fixed weight assignment, added loops enhancement, fixed data structure
Added missing doc blocks
Added support for Pub/Sub in MultiDBClient
Refactored configuration
Refactored failure detector
Refactored retry logic
Added scenario tests
Added pybreaker optional dependency
Added pybreaker to dev dependencies
Rename tests directory
Added scenario tests for Pipeline and Transaction
Added handling of ConnectionRefusedError, added timeouts so cluster could recover
Increased timeouts
Refactored integration tests
Added scenario tests for Pub/Sub
Updated healthcheck retry
Increased timeout to avoid unprepared state before tests
Added backoff retry and changed timeouts
Added retry for healthchecks to avoid fluctuations
Changed retry configuration for healthchecks
Fixed property name
Added check for thread results
Added MultiDbClient support with OSS Cluster API
Removed database statuses
Increased test timeouts
Increased retry timeout
Increased timeout retries
Updated base threshold for retries
Fixed flacky tests
Added missing docblocks

Added LagAwareHealthcheck
Added testing for LagAwareHealthCheck
Fixed timeouts
Added lag tollerance parameter
Decreased messages_count due to increased timeouts
Added docblocks
Added missing type hints
Fixed url
Refactored tests, URL and cluster support
Use primary node to send an API request
Added comment about RE bug
Moved None type to the beginning
Added health_check_url property to Database class

Extract additional interfaces and abstract classes
Added base async components
Added command executor
Added recurring background tasks with event loop only
Added MultiDBClient
Added scenario and config tests
Update redis/asyncio/multidb/healthcheck.py

Co-authored-by: Copilot 175728472+Copilot@users.noreply.github.com

Update tests/test_asyncio/test_scenario/test_active_active.py

Co-authored-by: Copilot 175728472+Copilot@users.noreply.github.com

Extract additional interfaces and abstract classes
Added base async components
Added command executor
Added recurring background tasks with event loop only
Added MultiDBClient
Added scenario and config tests
Added pipeline and transaction support for MultiDBClient
Updated scenario tests to check failover

Extract additional interfaces and abstract classes
Added base async components
Added command executor
Added recurring background tasks with event loop only
Added MultiDBClient
Added scenario and config tests
Added pipeline and transaction support for MultiDBClient
Added pub/sub support for MultiDBClient
Added check for couroutines methods for pub/sub

Extract additional interfaces and abstract classes
Added base async components
Added command executor
Added recurring background tasks with event loop only
Added MultiDBClient
Added scenario and config tests
Added pipeline and transaction support for MultiDBClient
Added pub/sub support for MultiDBClient
Added check for couroutines methods for pub/sub
Added OSS Cluster API support for MultiDBCLient
Added support for Lag-Aware Healthcheck and OSS Cluster API
Increased timeouts between tests
Fixed space

Extract additional interfaces and abstract classes
Added base async components
Added command executor
Added recurring background tasks with event loop only
Added MultiDBClient
Added scenario and config tests
Added pipeline and transaction support for MultiDBClient
Added pub/sub support for MultiDBClient
Added check for couroutines methods for pub/sub
Added OSS Cluster API support for MultiDBCLient
Added support for Lag-Aware Healthcheck and OSS Cluster API
Increased timeouts between tests
[Sync] Refactored healthcheck
[Async] Refactored healthcheck
[Sync] Refactored Failover Strategy
[Async] Refactored Failover Strategy
Changed default values according to a design doc
[Async] Added Strategy Executor
[Sync] Added Strategy Executor
Apply comments

Added Active-Active documentation page
Added documentation for Active-Active
Refactored docs
Refactored pipeline and transaction section
Updated docs
Extended list of words
Re-write documentation
Fixed spelling
Update docs/multi_database.rst

Co-authored-by: Elena Kolevska elena-kolevska@users.noreply.github.com

Apply suggested comments
Fixed spelling
Update docs/multi_database.rst

Co-authored-by: Elena Kolevska elena-kolevska@users.noreply.github.com

Update docs/multi_database.rst

Co-authored-by: Elena Kolevska elena-kolevska@users.noreply.github.com

Update docs/multi_database.rst

Co-authored-by: Elena Kolevska elena-kolevska@users.noreply.github.com