Skip to content

22 - numpy Bulk I/O Workflow

View source code

Every path in this example crosses the Python/Java boundary once per batch instead of once per value. Per-row FFI calls cap document ingest around 30k rows/s regardless of engine speed; the batched paths below reach hundreds of thousands of rows or points per second on the same machine.

It covers:

  • bulk document ingest with Database.insert_many() (rows serialized as one JSON batch, looped Java-side), in both transactional-batch and async parallel-writer modes. Each bucket is owned by one async writer, so the parallel mode runs on a type created with as many buckets as the executor's parallel level (async_executor().get_parallel_level(), default cores - 1; ArcadeData/arcadedb#8478). commit_every does not apply to that mode, and it raises ArcadeDBError after the load if the writers rejected any record. See insert_many
  • time-series ingest straight from numpy arrays via AsyncExecutor.append_samples() - timestamps and numeric field columns cross the boundary as one buffer copy per column
  • time-bucketed aggregation over the native TIMESERIES type (ts.timeBucket with a bounded WHERE ts BETWEEN range)
  • columnar export with to_columns(): scalar columns come back as 1-D numpy arrays and embedding columns as one contiguous 2-D float32 array, ready for scikit-learn or faiss without per-row conversion
  • the same export through to_arrow(), which reads the same bridge buffer but keeps the validity bitmap. A nullable INTEGER stays int64 instead of widening to float64 with NaN holes, and a nullable BOOLEAN stays bool instead of degrading to a Python list. That section needs pyarrow (pip install 'arcadedb-embedded[arrow]') and skips itself without it.

Run

From bindings/python/examples:

python 22_numpy_bulk_io.py
# reduced scale:
python 22_numpy_bulk_io.py --rows 50000 --points 100000 --vectors 5000

The database is created under ./my_test_databases/numpy_bulk_io (--db-path).