New LSMVector index using JVector open source vector engine by lvca · Pull Request #2816 · ArcadeData/arcadedb (original) (raw)

added 11 commits

November 18, 2025 20:11

…internal map (with threadId)

fix: error after compaction

[ gemini-code-assist[bot] ](/apps/gemini-code-assist)

Initial plan
Fix ComparableVector to maintain Comparable contract

Co-authored-by: lvca 312606+lvca@users.noreply.github.com

Co-authored-by: copilot-swe-agent[bot] 198982749+Copilot@users.noreply.github.com Co-authored-by: lvca 312606+lvca@users.noreply.github.com

lvca and others added 17 commits

November 22, 2025 11:08

…internal map (with threadId)

fix: error after compaction

Initial plan
Fix ComparableVector to maintain Comparable contract

Co-authored-by: lvca 312606+lvca@users.noreply.github.com

Co-authored-by: copilot-swe-agent[bot] 198982749+Copilot@users.noreply.github.com Co-authored-by: lvca 312606+lvca@users.noreply.github.com

Co-authored-by: Copilot 175728472+Copilot@users.noreply.github.com

…nce recalculation (#2820)

Initial plan
Add findNeighborsFromVector method to LSMVectorIndex to return scores directly

Co-authored-by: lvca 312606+lvca@users.noreply.github.com

Remove test artifacts and update .gitignore

Co-authored-by: lvca 312606+lvca@users.noreply.github.com

Co-authored-by: copilot-swe-agent[bot] 198982749+Copilot@users.noreply.github.com Co-authored-by: lvca 312606+lvca@users.noreply.github.com

Initial plan
Add mutable byte indicator to vector index pages

Added mutable flag byte at offset 8 in page header (after offsetFreeContent and numberOfEntries)
New pages are created with mutable=1 (actively being written to)
Pages are marked as immutable (mutable=0) when they become full and a new page is created
Updated findLastImmutablePage() to scan from end backwards and stop at first immutable page
Updated all page reading/writing code to account for the mutable byte in header
All vector index tests passing

Co-authored-by: lvca 312606+lvca@users.noreply.github.com

Remove test database files and update .gitignore

Co-authored-by: lvca 312606+lvca@users.noreply.github.com

Add constants for page header offsets to improve maintainability

Added OFFSET_FREE_CONTENT, OFFSET_NUM_ENTRIES, OFFSET_MUTABLE, and HEADER_BASE_SIZE constants
Replaced magic numbers throughout the code with named constants
Makes the code more maintainable and self-documenting

Co-authored-by: lvca 312606+lvca@users.noreply.github.com

Update comments to reference constants instead of hardcoded offsets

Co-authored-by: lvca 312606+lvca@users.noreply.github.com

Co-authored-by: copilot-swe-agent[bot] 198982749+Copilot@users.noreply.github.com Co-authored-by: lvca 312606+lvca@users.noreply.github.com Co-authored-by: Luca Garulli lvca@users.noreply.github.com

Initial plan
Make ID property configurable in LSMVectorIndex

Co-authored-by: lvca 312606+lvca@users.noreply.github.com

Remove test database files and add to .gitignore

Co-authored-by: lvca 312606+lvca@users.noreply.github.com

Improve documentation for metadata JSON configuration

Co-authored-by: lvca 312606+lvca@users.noreply.github.com

Co-authored-by: copilot-swe-agent[bot] 198982749+Copilot@users.noreply.github.com Co-authored-by: lvca 312606+lvca@users.noreply.github.com Co-authored-by: Luca Garulli lvca@users.noreply.github.com

lvca and others added 4 commits

November 22, 2025 11:16

…text.java

Co-authored-by: Copilot 175728472+Copilot@users.noreply.github.com

lvca changed the title~~JVector integration~~ New LSMVector index using JVector open source vector engine

Nov 22, 2025

robfrank linked an issue

Nov 25, 2025

that may beclosed by this pull request

4 tasks

This was referenced

Dec 9, 2025

robfrank pushed a commit that referenced this pull request

Feb 11, 2026

First version with jvector
Implemented compaction of vector indexes
Added test cases
Fixed compilation problems
Fixed test cases, now all pass
Refactor vector index using the transaction index changes instead of internal map (with threadId)
feat: integrated new vector index with the database import command
Supported lsmvector in vectorNeighbors() sql function
Upgraded to jvector 4.0.0-rc.6
Update LSMVectorIndexCompacted.java

fix: error after compaction

Fix ComparableVector Comparable contract violation (#2817)
Initial plan
Fix ComparableVector to maintain Comparable contract

Co-authored-by: lvca 312606+lvca@users.noreply.github.com

Co-authored-by: copilot-swe-agent[bot] 198982749+Copilot@users.noreply.github.com Co-authored-by: lvca 312606+lvca@users.noreply.github.com

Apply suggestion from @Copilot

Co-authored-by: Copilot 175728472+Copilot@users.noreply.github.com

Return similarity scores from LSMVectorIndex to avoid redundant distance recalculation (#2820)
Initial plan
Add findNeighborsFromVector method to LSMVectorIndex to return scores directly

Co-authored-by: lvca 312606+lvca@users.noreply.github.com

Remove test artifacts and update .gitignore

Co-authored-by: lvca 312606+lvca@users.noreply.github.com

Co-authored-by: copilot-swe-agent[bot] 198982749+Copilot@users.noreply.github.com Co-authored-by: lvca 312606+lvca@users.noreply.github.com

Add mutable flag to vector index pages for safe compaction (#2819)
Initial plan
Add mutable byte indicator to vector index pages

Added mutable flag byte at offset 8 in page header (after offsetFreeContent and numberOfEntries)
New pages are created with mutable=1 (actively being written to)
Pages are marked as immutable (mutable=0) when they become full and a new page is created
Updated findLastImmutablePage() to scan from end backwards and stop at first immutable page
Updated all page reading/writing code to account for the mutable byte in header
All vector index tests passing

Co-authored-by: lvca 312606+lvca@users.noreply.github.com

Remove test database files and update .gitignore

Co-authored-by: lvca 312606+lvca@users.noreply.github.com

Add constants for page header offsets to improve maintainability

Added OFFSET_FREE_CONTENT, OFFSET_NUM_ENTRIES, OFFSET_MUTABLE, and HEADER_BASE_SIZE constants
Replaced magic numbers throughout the code with named constants
Makes the code more maintainable and self-documenting

Co-authored-by: lvca 312606+lvca@users.noreply.github.com

Update comments to reference constants instead of hardcoded offsets

Co-authored-by: lvca 312606+lvca@users.noreply.github.com

Co-authored-by: copilot-swe-agent[bot] 198982749+Copilot@users.noreply.github.com Co-authored-by: lvca 312606+lvca@users.noreply.github.com Co-authored-by: Luca Garulli lvca@users.noreply.github.com

Make LSMVectorIndex ID property configurable (#2818)
Initial plan
Make ID property configurable in LSMVectorIndex

Co-authored-by: lvca 312606+lvca@users.noreply.github.com

Remove test database files and add to .gitignore

Co-authored-by: lvca 312606+lvca@users.noreply.github.com

Improve documentation for metadata JSON configuration

Co-authored-by: lvca 312606+lvca@users.noreply.github.com

Co-authored-by: copilot-swe-agent[bot] 198982749+Copilot@users.noreply.github.com Co-authored-by: lvca 312606+lvca@users.noreply.github.com Co-authored-by: Luca Garulli lvca@users.noreply.github.com

First version with jvector
Implemented compaction of vector indexes
Added test cases
Fixed compilation problems
Fixed test cases, now all pass
Refactor vector index using the transaction index changes instead of internal map (with threadId)
feat: integrated new vector index with the database import command
Supported lsmvector in vectorNeighbors() sql function
Upgraded to jvector 4.0.0-rc.6
Update LSMVectorIndexCompacted.java

fix: error after compaction

Fix ComparableVector Comparable contract violation (#2817)
Initial plan
Fix ComparableVector to maintain Comparable contract

Co-authored-by: lvca 312606+lvca@users.noreply.github.com

Co-authored-by: copilot-swe-agent[bot] 198982749+Copilot@users.noreply.github.com Co-authored-by: lvca 312606+lvca@users.noreply.github.com

Apply suggestion from @Copilot

Co-authored-by: Copilot 175728472+Copilot@users.noreply.github.com

Return similarity scores from LSMVectorIndex to avoid redundant distance recalculation (#2820)
Initial plan
Add findNeighborsFromVector method to LSMVectorIndex to return scores directly

Co-authored-by: lvca 312606+lvca@users.noreply.github.com

Remove test artifacts and update .gitignore

Co-authored-by: lvca 312606+lvca@users.noreply.github.com

Co-authored-by: copilot-swe-agent[bot] 198982749+Copilot@users.noreply.github.com Co-authored-by: lvca 312606+lvca@users.noreply.github.com

Add mutable flag to vector index pages for safe compaction (#2819)
Initial plan
Add mutable byte indicator to vector index pages

Added mutable flag byte at offset 8 in page header (after offsetFreeContent and numberOfEntries)
New pages are created with mutable=1 (actively being written to)
Pages are marked as immutable (mutable=0) when they become full and a new page is created
Updated findLastImmutablePage() to scan from end backwards and stop at first immutable page
Updated all page reading/writing code to account for the mutable byte in header
All vector index tests passing

Co-authored-by: lvca 312606+lvca@users.noreply.github.com

Remove test database files and update .gitignore

Co-authored-by: lvca 312606+lvca@users.noreply.github.com

Add constants for page header offsets to improve maintainability

Added OFFSET_FREE_CONTENT, OFFSET_NUM_ENTRIES, OFFSET_MUTABLE, and HEADER_BASE_SIZE constants
Replaced magic numbers throughout the code with named constants
Makes the code more maintainable and self-documenting

Co-authored-by: lvca 312606+lvca@users.noreply.github.com

Update comments to reference constants instead of hardcoded offsets

Co-authored-by: lvca 312606+lvca@users.noreply.github.com

Co-authored-by: copilot-swe-agent[bot] 198982749+Copilot@users.noreply.github.com Co-authored-by: lvca 312606+lvca@users.noreply.github.com Co-authored-by: Luca Garulli lvca@users.noreply.github.com

Make LSMVectorIndex ID property configurable (#2818)
Initial plan
Make ID property configurable in LSMVectorIndex

Co-authored-by: lvca 312606+lvca@users.noreply.github.com

Remove test database files and add to .gitignore

Co-authored-by: lvca 312606+lvca@users.noreply.github.com

Improve documentation for metadata JSON configuration

Co-authored-by: lvca 312606+lvca@users.noreply.github.com

Co-authored-by: copilot-swe-agent[bot] 198982749+Copilot@users.noreply.github.com Co-authored-by: lvca 312606+lvca@users.noreply.github.com Co-authored-by: Luca Garulli lvca@users.noreply.github.com

fix pre-commit
Update engine/src/main/java/com/arcadedb/database/TransactionIndexContext.java

Co-authored-by: Copilot 175728472+Copilot@users.noreply.github.com

Fixed compaction
test: fixed test

Co-authored-by: Copilot 198982749+Copilot@users.noreply.github.com Co-authored-by: lvca 312606+lvca@users.noreply.github.com Co-authored-by: Copilot 175728472+Copilot@users.noreply.github.com Co-authored-by: Roberto Franchini ro.franchini@gmail.com

(cherry picked from commit c470e6d)

This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters

[ Show hidden characters]({{ revealButtonHref }})