Concurrency Tests¶

View source code

There are 5 tests demonstrating file locking, thread safety, sequential access, concurrent limitation, and a mixed OLTP-style workload.

Key Insight¶

❌ Multiple processes cannot access same database (file lock prevents)
✅ Multiple threads can (thread-safe within same process)
✅ For true multi-process access, run the standalone ArcadeDB server

Test Cases¶

1) file lock mechanism¶

Opens database, checks for database.lck file, closes, verifies lock is released.

2) thread safety¶

Creates database with 20 Person records, spawns 4 threads querying disjoint id ranges (0–4, 5–9, 10–14, 15–19) concurrently, asserts all threads complete and find correct counts.

3) sequential access¶

Sequence: create+insert → close → reopen+query (1 record) → close → reopen+insert+query (2 records). Verifies data persists and each step works.

4) concurrent access limitation¶

Attempts concurrent access from same process; documents that a second handle to locked DB raises ArcadeDBError.

5) OLTP mixed workload (multi-thread)¶

Runs a mixed read/write workload across multiple threads, measures basic latency/throughput, and includes retry/backoff for concurrent modifications.

Architecture¶

Single-process (embedded): Thread-safe; file lock prevents other processes
Multi-process: Requires the standalone ArcadeDB server (HTTP API); embedded access remains file-locked

Overview¶

The Key Question

"Can multiple Python instances access the same database?"

Short Answer:

❌ Multiple processes cannot (file lock prevents it)
✅ Multiple threads can (thread-safe within same process)
✅ For true multi-process access, run the standalone ArcadeDB server

These tests demonstrate:

File locking mechanism (OS-level locks)
Thread safety within a single process
Sequential access patterns (open → close → reopen)
Multi-process limitations and solutions

Why This Matters¶

Understanding ArcadeDB's concurrency model is critical for:

Deployment architecture: Knowing when to use embedded vs a standalone server
Multi-process applications: Understanding limitations and workarounds
Thread safety: Confidently using threads with shared database access
Performance optimization: Choosing the right access pattern

Test Cases¶

1. File Lock Mechanism¶

Test: test_file_lock_mechanism

Demonstrates that ArcadeDB uses OS-level file locks to prevent concurrent access from multiple processes.

import arcadedb_embedded as arcadedb
from arcadedb_embedded.exceptions import ArcadeDBError

# Process 1: Create and hold lock
db1 = arcadedb.create_database("./test_db")

# Process 2: Try to open same database (this would be in another Python process)
try:
    db2 = arcadedb.open_database("./test_db")
    # This will fail with LockException
except ArcadeDBError as e:
    print(f"Expected error: {e}")
    # Error message contains: "LockException" or "already locked"

db1.close()

What happens:

First database open acquires an OS-level file lock
Second attempt to open fails immediately
Lock is released when database is closed
Java throws LockException, wrapped as ArcadeDBError in Python

Why it exists:

Prevents database corruption from concurrent writes
Ensures data consistency
Standard practice for embedded databases (SQLite does the same)

Multi-Process Access

You cannot open the same database from multiple Python processes simultaneously.

# ❌ This WILL NOT work
# process_1.py
db = arcadedb.create_database("./mydb")  # Locks the database

# process_2.py (running simultaneously)
db = arcadedb.open_database("./mydb")    # FAILS with LockException!

Solution: Use the standalone ArcadeDB server for multi-process access.

2. Thread-Safe Operations¶

Test: test_thread_safety

Demonstrates that multiple threads can safely access the same database instance.

import threading
import arcadedb_embedded as arcadedb

# Create database
db = arcadedb.create_database("./test_db")
db.command("sql", "CREATE DOCUMENT TYPE Counter")

# Insert initial record
with db.transaction():
    counter = db.new_document("Counter")
    counter.set("value", 0).save()

def increment_counter(thread_id, iterations):
    """Each thread increments the counter"""
    for i in range(iterations):
        with db.transaction():
            # Read current value
            result = db.query("sql", "SELECT FROM Counter")
            current = list(result)[0].get("value")

            # Increment
            db.command("sql", f"UPDATE Counter SET value = {current + 1}")

# Create multiple threads
threads = []
num_threads = 5
iterations_per_thread = 10

for i in range(num_threads):
    thread = threading.Thread(
        target=increment_counter,
        args=(i, iterations_per_thread)
    )
    threads.append(thread)
    thread.start()

# Wait for all threads to complete
for thread in threads:
    thread.join()

# Verify final count
result = db.query("sql", "SELECT FROM Counter")
final_value = list(result)[0].get("value")

# Should be num_threads * iterations_per_thread
assert final_value == 50

db.close()

What it tests:

Multiple threads accessing the same Database instance
Concurrent read/write operations
Transaction isolation between threads
No race conditions or data corruption

Key insight:

✅ ArcadeDB is thread-safe! Multiple threads in the same process can safely share a database instance.

Thread Safety

The Java ArcadeDB engine handles internal synchronization, so you can confidently use the same database object across multiple threads.

# ✅ This WORKS perfectly
db = arcadedb.create_database("./mydb")

def worker():
    with db.transaction():
        rec = db.new_document("MyType")
        rec.set("data", "value").save()

# Multiple threads share the same db instance
threads = [Thread(target=worker) for _ in range(10)]
for t in threads:
    t.start()

Best practice:

Use a single database instance shared across threads:

# Good: Share one database instance
db = arcadedb.create_database("./mydb")
db.command("sql", "CREATE DOCUMENT TYPE MyType")

def thread_worker():
    result = db.query("sql", "SELECT FROM MyType")
    # Process results...

threads = [Thread(target=thread_worker) for _ in range(10)]

Don't create separate database instances per thread (unnecessary overhead):

# Avoid: Don't do this
def thread_worker():
    db = arcadedb.open_database("./mydb")  # ❌ Creates separate instance
    result = db.query("sql", "SELECT FROM MyType")
    db.close()

3. Sequential Access Pattern¶

Test: test_sequential_access

Demonstrates that a database can be closed and reopened sequentially.

import arcadedb_embedded as arcadedb

# First: Create and populate
db1 = arcadedb.create_database("./test_db")
db1.command("sql", "CREATE DOCUMENT TYPE Person")

with db1.transaction():
    person = db1.new_document("Person")
    person.set("name", "Alice").set("age", 30).save()

# Close to release file lock
db1.close()

# Second: Reopen and verify data persisted
db2 = arcadedb.open_database("./test_db")

result = db2.query("sql", "SELECT FROM Person WHERE name = 'Alice'")
person = list(result)[0]

assert person.get("name") == "Alice"
assert person.get("age") == 30

db2.close()

What it tests:

Database persistence across close/reopen cycles
File lock release on close()
Data integrity after reopening
Proper resource cleanup

Common pattern:

This is useful for:

Batch processing: Open → process → close → repeat
Migration to a standalone server: Create & populate → close → move into the server's databases directory
Temporary exclusive access: Open → do work → close (release lock)

Important:

# Pattern: Create → Use → Close → Reopen
db = arcadedb.create_database("./mydb")
# ... do work ...
db.close()  # ⚠️ Must close to release lock

# Now someone else (or same process) can open it
db2 = arcadedb.open_database("./mydb")  # ✅ Works!

4. Multi-Process Limitations¶

Test: test_concurrent_access_limitation

Explicitly demonstrates why multiple processes cannot access the same database directly, and shows the solution.

import arcadedb_embedded as arcadedb
from arcadedb_embedded.exceptions import ArcadeDBError
import subprocess
import sys

# Create test database
db = arcadedb.create_database("./test_db")
db.command("sql", "CREATE DOCUMENT TYPE Data")
with db.transaction():
    doc = db.new_document("Data")
    doc.set("value", "test").save()
db.close()

# Attempt to access from subprocess
script = """
import arcadedb_embedded as arcadedb
try:
    db = arcadedb.open_database("./test_db")
    print("SUCCESS")
    db.close()
except Exception as e:
    print(f"FAILED: {e}")
"""

# This subprocess will fail if main process holds lock
result = subprocess.run(
    [sys.executable, "-c", script],
    capture_output=True,
    text=True
)

print("Subprocess output:", result.stdout)

Why this fails:

Process 1 (main):     db1 = open("./mydb")  🔒 Lock acquired
                      |
                      | (both processes running)
                      |
Process 2 (subprocess): db2 = open("./mydb")  ❌ LockException!

The solution: run the official ArcadeDB server

# One standalone server owns the databases
docker run -d -p 2480:2480 -p 2424:2424 \
  -e JAVA_OPTS="-Darcadedb.server.rootPassword=playwithdata" \
  arcadedata/arcadedb:latest

# All processes access it over HTTP
import requests
response = requests.post(
    'http://localhost:2480/api/v1/query/mydb',
    json={
        'language': 'sql',
        'command': 'SELECT FROM MyType'
    },
    auth=('root', 'playwithdata')
)

Key insight:

Multi-Process Architecture

This package is embedded-only (single process owns the files). For multi-process applications, run the standalone ArcadeDB server and have every process talk to it over HTTP.

Summary Table¶

Scenario	Supported?	Notes
Multiple threads, same process	✅ Yes	Thread-safe, share database instance
Sequential: open → close → reopen	✅ Yes	Must close to release lock
Multiple processes, embedded mode	❌ No	File lock prevents concurrent access
Multiple processes via standalone ArcadeDB server	✅ Yes	All processes use its HTTP API

Running These Tests¶

# Run all concurrency tests
pytest tests/test_concurrency.py -v

# Run specific test
pytest tests/test_concurrency.py::test_file_lock_mechanism -v
pytest tests/test_concurrency.py::test_thread_safety -v

# Run with output to see details
pytest tests/test_concurrency.py -v -s

Best Practices¶

✅ DO: Use Threads for Parallelism¶

db = arcadedb.create_database("./mydb")

def worker(worker_id):
    with db.transaction():
        rec = db.new_document("Data")
        rec.set("worker", worker_id).save()

threads = [Thread(target=worker, args=(i,)) for i in range(10)]
for t in threads:
    t.start()
for t in threads:
    t.join()

db.close()

✅ DO: Use the Standalone Server for Multi-Process¶

# One standalone ArcadeDB server owns the databases;
# every process connects over its HTTP API
docker run -d -p 2480:2480 arcadedata/arcadedb:latest

✅ DO: Close Between Sequential Opens¶

# Process 1
db = arcadedb.create_database("./mydb")
# ... work ...
db.close()  # Release lock

# Process 2 (later, or different script)
db = arcadedb.open_database("./mydb")  # Works!

❌ DON'T: Try Concurrent Process Access¶

# ❌ This will fail
# script1.py
db1 = arcadedb.create_database("./mydb")
# keeps running...

# script2.py (simultaneously)
db2 = arcadedb.open_database("./mydb")  # LockException!

❌ DON'T: Create Multiple DB Instances in Threads¶

# ❌ Inefficient - creates unnecessary instances
def worker():
    db = arcadedb.open_database("./mydb")  # Separate instance
    # ... work ...
    db.close()

# ✅ Better - share one instance
db = arcadedb.open_database("./mydb")

def worker():
    # Use shared db instance
    result = db.query("sql", "SELECT ...")

Access Methods - Embedded vs standalone server
Database API - Database class reference

Troubleshooting¶

"Database is locked" Error¶

# Error: LockException: database ./mydb is already locked

Cause: Another process (or unclosed instance) holds the file lock.

Solutions:

Close the other database instance: db.close()
Check for zombie processes holding locks
Use server mode for multi-process access
Restart if lock file is orphaned

Thread Safety Concerns¶

Q: Is it safe to share a database instance across threads?

A: Yes! ArcadeDB handles internal synchronization. You can safely share a Database instance across multiple threads.

Q: Do I need to synchronize access to the database?

A: No, unless you need application-level coordination (e.g., ensuring specific order of operations). The database itself is thread-safe.

Performance with Threads¶

Q: Should I use one database instance or multiple?

A: Use one shared instance. Creating multiple instances adds overhead with no benefit since they'll all access the same underlying database through Java.

Concurrency Tests¶

Key Insight¶

Test Cases¶

1) file lock mechanism¶

2) thread safety¶

3) sequential access¶

4) concurrent access limitation¶

5) OLTP mixed workload (multi-thread)¶

Architecture¶

Overview¶

Why This Matters¶

Test Cases¶

1. File Lock Mechanism¶

2. Thread-Safe Operations¶

3. Sequential Access Pattern¶

4. Multi-Process Limitations¶

Summary Table¶

Running These Tests¶

Best Practices¶

✅ DO: Use Threads for Parallelism¶

✅ DO: Use the Standalone Server for Multi-Process¶

✅ DO: Close Between Sequential Opens¶

❌ DON'T: Try Concurrent Process Access¶

❌ DON'T: Create Multiple DB Instances in Threads¶

Related Documentation¶

Troubleshooting¶

"Database is locked" Error¶

Thread Safety Concerns¶

Performance with Threads¶

Further Reading¶