Details
-
Bug
-
Status: Closed
-
Blocker
-
Resolution: Fixed
-
6.5.0, 6.0.3
-
Untriaged
-
Unknown
-
CX Sprint 185
Description
As observed in a customer Kubernetes environment, the CC node was recreated with the existing storage, but assigned a new IP address. Due to network address caching, one of the existing NCs was perpetually unable to connect to the CC as it did not realize the IP address update until the driver could be restarted.
We should set networkaddress.cache.ttl to a reasonable value, so that we are protected from this situation. e.g. something <= 5 minutes
The above suggestion is insufficient, due to caching above the name service which is also a factor.
Attachments
Issue Links
- is parent task of
-
MB-37834 [CX] Add IP address update tests (see MB-37811)
-
- Closed
-
- links to
Activity
| Field | Original Value | New Value |
|---|---|---|
| Link | This issue causes CBSE-7909 [ CBSE-7909 ] |
| Labels | 6.5.1-candidate 6.x-candidate | 6.5.1-candidate 6.x-candidate kubernetes |
| Rank | Ranked higher |
| Rank | Ranked higher |
| Sprint | CX Sprint 185 [ 983 ] |
| Rank | Ranked lower |
| Labels | 6.5.1-candidate 6.x-candidate kubernetes | 6.5.1-candidate 6.x-candidate kubernetes releasenote |
| Status | Open [ 1 ] | In Progress [ 3 ] |
| Labels | 6.5.1-candidate 6.x-candidate kubernetes releasenote | 6.0.5-candidate 6.5.1-candidate 6.x-candidate kubernetes releasenote |
| Labels | 6.0.5-candidate 6.5.1-candidate 6.x-candidate kubernetes releasenote | 6.0.5-candidate 6.5.1-candidate 6.x-candidate kubernetes releasenote triaged |
| Labels | 6.0.5-candidate 6.5.1-candidate 6.x-candidate kubernetes releasenote triaged | 6.0.5-candidate 6.5.1-candidate 6.x-candidate DBaaS kubernetes releasenote triaged |
| Affects Version/s | Mad-Hatter [ 15037 ] | |
| Affects Version/s | 6.5.0 [ 16624 ] |
| Summary | [CX] Infinite DNS lookup address caching breaks clusters on IP reassignment | [CX] Analytics cluster down on IP reassignment |
| Description |
As observed in a customer Kubernetes environment, the CC node was recreated with the existing storage, but assigned a new IP address. Due to network address caching, one of the existing NCs was perpetually unable to connect to the CC as it did not realize the IP address update until the driver could be restarted.
We should set {{networkaddress.cache.ttl}} to a reasonable value, so that we are protected from this situation. e.g. something <= 5 minutes |
As observed in a customer Kubernetes environment, the CC node was recreated with the existing storage, but assigned a new IP address. Due to network address caching, one of the existing NCs was perpetually unable to connect to the CC as it did not realize the IP address update until the driver could be restarted.
-We should set {{networkaddress.cache.ttl}} to a reasonable value, so that we are protected from this situation. e.g. something <= 5 minutes- The above suggestion is insufficient, due to caching above the name service which is also a factor. |
| Link | This issue blocks MB-37192 [ MB-37192 ] |
| Remote Link | This issue links to "AsterixDB Gerrit Review (Web Link)" [ 19202 ] |
| Resolution | Fixed [ 1 ] | |
| Status | In Progress [ 3 ] | Resolved [ 5 ] |
| Labels | 6.0.5-candidate 6.5.1-candidate 6.x-candidate DBaaS kubernetes releasenote triaged | 6.0.5-candidate DBaaS approved-for-6.5.1 kubernetes releasenote triaged |
| Remote Link | This issue links to "AsterixDB Gerrit Review (Web Link)" [ 19223 ] |
| Assignee | Michael Blow [ michael.blow ] | Arunkumar Senthilnathan [ arunkumar ] |
| Status | Resolved [ 5 ] | Closed [ 6 ] |
| Labels | 6.0.5-candidate DBaaS approved-for-6.5.1 kubernetes releasenote triaged | 6.0.6-candidate DBaaS approved-for-6.5.1 kubernetes releasenote triaged |