简介:
Quick Start:
http://www.lmdb.tech/doc/starting.html
全部接口介绍:
http://www.lmdb.tech/doc/group__mdb.html
The entire database is exposed in a memory map, and all data fetches return data directly from the mapped memory, so no malloc’s or memcpy’s occur during data fetches.
The memory map can be used as a read-only or read-write map. It is read-only by default as this provides total immunity to corruption. Using read-write mode offers much higher write performance, but adds the possibility for stray application writes thru pointers to silently corrupt the database. Of course if your application code is known to be bug-free (…) then this is not an issue.
Quick Start
Getting Started
Everything starts with an environment, created by mdb_env_create(). Once created, this environment must also be opened with mdb_env_open().
mdb_env_open() gets passed a name which is interpreted as a directory path. Note that this directory must exist already, it is not created for you. Within that directory, a lock file and a storage file will be generated. If you don’t want to use a directory, you can pass the MDB_NOSUBDIR option, in which case the path you provided is used directly as the data file, and another file with a “-lock” suffix added will be used for the lock file.
Once the environment is open, a transaction can be created within it using mdb_txn_begin(). Transactions may be read-write or read-only, and read-write transactions may be nested. A transaction must only be used by one thread at a time. Transactions are always required, even for read-only access. The transaction provides a consistent view of the data.
Once a transaction has been created, a database can be opened within it using mdb_dbi_open(). If only one database will ever be used in the environment, a NULL can be passed as the database name. For named databases, the MDB_CREATE flag must be used to create the database if it doesn’t already exist. Also, mdb_env_set_maxdbs() must be called after mdb_env_create() and before mdb_env_open() to set the maximum number of named databases you want to support. mdb_env_set_mapsize
Note: a single transaction can open multiple databases. Generally databases should only be opened once, by the first transaction in the process. After the first transaction completes, the database handles can freely be used by all subsequent transactions.
Within a transaction, mdb_get() and mdb_put() can store single key/value pairs if that is all you need to do (but see Cursors below if you want to do more).
A key/value pair is expressed as two MDB_val structures. This struct has two fields, mv_size
and mv_data
. The data is a void
pointer to an array of mv_size
bytes.
Because LMDB is very efficient (and usually zero-copy), the data returned in an MDB_val structure may be memory-mapped straight from disk. In other words look but do not touch (or free() for that matter). Once a transaction is closed, the values can no longer be used, so make a copy if you need to keep them after that.
Cursor
To do more powerful things, we must use a cursor.
Within the transaction, a cursor can be created with mdb_cursor_open(). With this cursor we can store/retrieve/delete (multiple) values using mdb_cursor_get(), mdb_cursor_put(), and mdb_cursor_del().
mdb_cursor_get() positions itself depending on the cursor operation requested, and for some operations, on the supplied key. For example, to list all key/value pairs in a database, use operation MDB_FIRST for the first call to mdb_cursor_get(), and MDB_NEXT on subsequent calls, until the end is hit.
To retrieve all keys starting from a specified key value, use MDB_SET. For more cursor operations, see the LMDB API docs.
When using mdb_cursor_put(), either the function will position the cursor for you based on the key, or you can use operation MDB_CURRENT to use the current position of the cursor. Note that key must then match the current position’s key.
Summarizing the Opening
So we have a cursor in a transaction which opened a database in an environment which is opened from a filesystem after it was separately created.
Or, we create an environment, open it from a filesystem, create a transaction within it, open a database within that transaction, and create a cursor within all of the above.
Got it?
Threads and Processes
LMDB uses POSIX locks on files, and these locks have issues if one process opens a file multiple times. Because of this, do not mdb_env_open() a file multiple times from a single process. Instead, share the LMDB environment that has opened the file across all threads.
Otherwise, if a single process opens the same environment multiple times, closing it once will remove all the locks held on it, and the other instances will be vulnerable to corruption from other processes.
Also note that a transaction is tied to one thread by default using Thread Local Storage. If you want to pass read-only transactions across threads, you can use the MDB_NOTLS option on the environment.
Transactions, Rollbacks, etc.
To actually get anything done, a transaction must be committed using mdb_txn_commit(). Alternatively, all of a transaction’s operations can be discarded using mdb_txn_abort().
In a read-only transaction, any cursors will not automatically be freed. In a read-write transaction, all cursors will be freed and must not be used again.
For read-only transactions, obviously there is nothing to commit to storage. The transaction still must eventually be aborted to close any database handle(s) opened in it, or committed to keep the database handles around for reuse in new transactions.
In addition, as long as a transaction is open, a consistent view of the database is kept alive, which requires storage. A read-only transaction that no longer requires this consistent view should be terminated (committed or aborted) when the view is no longer needed (but see below for an optimization).
There can be multiple simultaneously active read-only transactions but only one that can write. Once a single read-write transaction is opened, all further attempts to begin one will block until the first one is committed or aborted. This has no effect on read-only transactions, however, and they may continue to be opened at any time.
Duplicate Keys
mdb_get() and mdb_put() respectively have no and only some support for multiple key/value pairs with identical keys. If there are multiple values for a key, mdb_get() will only return the first value.
When multiple values for one key are required, pass the MDB_DUPSORT flag to mdb_dbi_open(). In an MDB_DUPSORT database, by default mdb_put() will not replace the value for a key if the key existed already. Instead it will add the new value to the key. In addition, mdb_del() will pay attention to the value field too, allowing for specific values of a key to be deleted.
Finally, additional cursor operations become available for traversing through and retrieving duplicate values.
Some Optimization
If you frequently begin and abort read-only transactions, as an optimization, it is possible to only reset and renew a transaction.
mdb_txn_reset() releases any old copies of data kept around for a read-only transaction. To reuse this reset transaction, call mdb_txn_renew() on it. Any cursors in this transaction must also be renewed using mdb_cursor_renew().
Note that mdb_txn_reset() is similar to mdb_txn_abort() and will close any databases you opened within the transaction.
To permanently free a transaction, reset or not, use mdb_txn_abort().
Cleaning Up
For read-only transactions, any cursors created within it must be closed using mdb_cursor_close().
It is very rarely necessary to close a database handle, and in general they should just be left open.
The Full API
The full LMDB API documentation lists further details, like how to:
- size a database (the default limits are intentionally small)
- drop and clean a database
- detect and report errors
- optimize (bulk) loading speed
- (temporarily) reduce robustness to gain even more speed
- gather statistics about the database
- define custom sort orders
Full API
lmdb很简单
Functions
Functions | |
---|---|
char * | mdb_version (int *major, int *minor, int *patch) |
Return the LMDB library version information. | |
char * | mdb_strerror (int err) |
Return a string describing a given error code. | |
env
Functions | |
---|---|
int | mdb_env_create (MDB_env **env) |
Create an LMDB environment handle. | |
int | mdb_env_open (MDB_env *env, const char *path, unsigned int flags, mdb_mode_t mode) |
Open an environment handle. | |
flags :MDB_FIXEDMAP use a fixed address for the mmap region. This flag must be specified when creating the environment, and is stored persistently in the environment. If successful, the memory map will always reside at the same virtual address and pointers used to reference data items in the database will be constant across multiple invocations. This option may not always work, depending on how the operating system has allocated memory to shared libraries and other uses. The feature is highly experimental. MDB_NOSUBDIR MDB_RDONLY Open the environment in read-only mode. No write operations will be allowed. LMDB will still modify the lock file - except on read-only filesystems, where LMDB does not use locks. MDB_WRITEMAP Use a writeable memory map unless MDB_RDONLY is set. This is faster and uses fewer mallocs, but loses protection from application bugs like wild pointer writes and other bad updates into the database. Incompatible with nested transactions. Do not mix processes with and without MDB_WRITEMAP on the same environment. This can defeat durability (mdb_env_sync etc). MDB_NOMETASYNC Flush system buffers to disk only once per transaction, omit the metadata flush. Defer that until the system flushes files to disk, or next non-MDB_RDONLY commit or mdb_env_sync(). This optimization maintains database integrity, but a system crash may undo the last committed transaction. I.e. it preserves the ACI (atomicity, consistency, isolation) but not D (durability) database property. This flag may be changed at any time using mdb_env_set_flags(). MDB_NOSYNC Don’t flush system buffers to disk when committing a transaction. This optimization means a system crash can corrupt the database or lose the last transactions if buffers are not yet flushed to disk. The risk is governed by how often the system flushes dirty buffers to disk and how often mdb_env_sync() is called. However, if the filesystem preserves write order and the MDB_WRITEMAP flag is not used, transactions exhibit ACI (atomicity, consistency, isolation) properties and only lose D (durability). I.e. database integrity is maintained, but a system crash may undo the final transactions. Note that (MDB_NOSYNC | MDB_WRITEMAP) leaves the system with no hint for when to write transactions to disk, unless mdb_env_sync() is called. (MDB_MAPASYNC | MDB_WRITEMAP) may be preferable. This flag may be changed at any time using mdb_env_set_flags(). MDB_MAPASYNC When using MDB_WRITEMAP, use asynchronous flushes to disk. As with MDB_NOSYNC, a system crash can then corrupt the database or lose the last transactions. Calling mdb_env_sync() ensures on-disk database integrity until next commit. This flag may be changed at any time using mdb_env_set_flags(). MDB_NOTLS Don’t use Thread-Local Storage. Tie reader locktable slots to MDB_txn objects instead of to threads. I.e. mdb_txn_reset() keeps the slot reseved for the MDB_txn object. A thread may use parallel read-only transactions. A read-only transaction may span threads if the user synchronizes its use. Applications that multiplex many user threads over individual OS threads need this option. Such an application must also serialize the write transactions in an OS thread, since LMDB’s write locking is unaware of the user threads. MDB_NOLOCK Don’t do any locking. If concurrent access is anticipated, the caller must manage all concurrency itself. For proper operation the caller must enforce single-writer semantics, and must ensure that no readers are using old transactions while a writer is active. The simplest approach is to use an exclusive lock so that no readers may be active at all when a writer begins. MDB_NORDAHEAD Turn off readahead. Most operating systems perform readahead on read requests by default. This option turns it off if the OS supports it. Turning it off may help random read performance when the DB is larger than RAM and system RAM is full. The option is not implemented on Windows. MDB_NOMEMINIT Don’t initialize malloc’d memory before writing to unused spaces in the data file. By default, memory for pages written to the data file is obtained using malloc. While these pages may be reused in subsequent transactions, freshly malloc’d pages will be initialized to zeroes before use. This avoids persisting leftover data from other code (that used the heap and subsequently freed the memory) into the data file. Note that many other system libraries may allocate and free memory from the heap for arbitrary uses. E.g., stdio may use the heap for file I/O buffers. This initialization step has a modest performance cost so some applications may want to disable it using this flag. This option can be a problem for applications which handle sensitive data like passwords, and it makes memory checkers like Valgrind noisy. This flag is not needed with MDB_WRITEMAP, which writes directly to the mmap instead of using malloc for pages. The initialization is also skipped if MDB_RESERVE is used; the caller is expected to overwrite all of the memory that was reserved in that case. This flag may be changed at any time using mdb_env_set_flags(). |
|
int | mdb_env_copy (MDB_env *env, const char *path) |
Copy an LMDB environment to the specified path. | |
int | mdb_env_copyfd (MDB_env *env, mdb_filehandle_t fd) |
Copy an LMDB environment to the specified file descriptor. | |
int | mdb_env_copy2 (MDB_env *env, const char *path, unsigned int flags) |
Copy an LMDB environment to the specified path, with options. | |
int | mdb_env_copyfd2 (MDB_env *env, mdb_filehandle_t fd, unsigned int flags) |
Copy an LMDB environment to the specified file descriptor, with options. | |
int | mdb_env_stat (MDB_env *env, MDB_stat *stat) |
Return statistics about the LMDB environment. | |
int | mdb_env_info (MDB_env *env, MDB_envinfo *stat) |
Return information about the LMDB environment. | |
int | mdb_env_sync (MDB_env *env, int force) |
Flush the data buffers to disk. | |
void | mdb_env_close (MDB_env *env) |
Close the environment and release the memory map. | |
int | mdb_env_set_flags (MDB_env *env, unsigned int flags, int onoff) |
Set environment flags. or to unset these flags | |
int | mdb_env_get_flags (MDB_env *env, unsigned int *flags) |
Get environment flags. | |
int | mdb_env_get_path (MDB_env *env, const char **path) |
Return the path that was used in mdb_env_open(). | |
int | mdb_env_get_fd (MDB_env *env, mdb_filehandle_t *fd) |
Return the filedescriptor for the given environment. | |
int | mdb_env_set_mapsize (MDB_env *env, size_t size) |
Set the size of the memory map to use for this environment. | |
The default is 10485760 bytes | |
int | mdb_env_set_maxreaders (MDB_env *env, unsigned int readers) |
Set the maximum number of threads/reader slots for the environment. | |
int | mdb_env_get_maxreaders (MDB_env *env, unsigned int *readers) |
Get the maximum number of threads/reader slots for the environment. | |
int | mdb_env_set_maxdbs (MDB_env *env, MDB_dbi dbs) |
Set the maximum number of named databases for the environment. | |
int | mdb_env_get_maxkeysize (MDB_env *env) |
Get the maximum size of keys and MDB_DUPSORT data we can write. | |
int | mdb_env_set_userctx (MDB_env *env, void *ctx) |
Set application information associated with the MDB_env. | |
void * | mdb_env_get_userctx (MDB_env *env) |
Get the application information associated with the MDB_env. | |
int | mdb_env_set_assert (MDB_env *env, MDB_assert_func *func) |
txn
Functions | |
---|---|
int | mdb_txn_begin (MDB_env *env, MDB_txn *parent, unsigned int flags, MDB_txn **txn) |
Create a transaction for use with the environment. | |
flags :MDB_RDONLY This transaction will not perform any write operations. |
|
MDB_env * | mdb_txn_env (MDB_txn *txn) |
Returns the transaction’s MDB_env. | |
size_t | mdb_txn_id (MDB_txn *txn) |
Return the transaction’s ID. | |
int | mdb_txn_commit (MDB_txn *txn) |
Commit all the operations of a transaction into the database. | |
void | mdb_txn_abort (MDB_txn *txn) |
Abandon all the operations of the transaction instead of saving them. | |
void | mdb_txn_reset (MDB_txn *txn) |
Reset a read-only transaction. | |
int | mdb_txn_renew (MDB_txn *txn) |
Renew a read-only transaction. | |
dbi
Functions | |
---|---|
int | mdb_dbi_open (MDB_txn *txn, const char *name, unsigned int flags, MDB_dbi *dbi) |
Open a database in the environment. | |
int | mdb_stat (MDB_txn *txn, MDB_dbi dbi, MDB_stat *stat) |
Retrieve statistics for a database. | |
int | mdb_dbi_flags (MDB_txn *txn, MDB_dbi dbi, unsigned int *flags) |
Retrieve the DB flags for a database handle. | |
void | mdb_dbi_close (MDB_env *env, MDB_dbi dbi) |
Close a database handle. Normally unnecessary. Use with care: | |
int | mdb_drop (MDB_txn *txn, MDB_dbi dbi, int del) |
Empty or delete+close a database. | |
int | mdb_set_compare (MDB_txn *txn, MDB_dbi dbi, MDB_cmp_func *cmp) |
Set a custom key comparison function for a database. | |
int | mdb_set_dupsort (MDB_txn *txn, MDB_dbi dbi, MDB_cmp_func *cmp) |
Set a custom data comparison function for a MDB_DUPSORT database. | |
int | mdb_set_relfunc (MDB_txn *txn, MDB_dbi dbi, MDB_rel_func *rel) |
Set a relocation function for a MDB_FIXEDMAP database. | |
int | mdb_set_relctx (MDB_txn *txn, MDB_dbi dbi, void *ctx) |
Set a context pointer for a MDB_FIXEDMAP database’s relocation function. | |
read/write on dbi
Functions | |
---|---|
int | mdb_get (MDB_txn *txn, MDB_dbi dbi, MDB_val *key, MDB_val *data) |
Get items from a database. | |
int | mdb_put (MDB_txn *txn, MDB_dbi dbi, MDB_val *key, MDB_val *data, unsigned int flags) |
Store items into a database. | |
int | mdb_del (MDB_txn *txn, MDB_dbi dbi, MDB_val *key, MDB_val *data) |
Delete items from a database. | |
int | mdb_cursor_open (MDB_txn *txn, MDB_dbi dbi, MDB_cursor **cursor) |
Create a cursor handle. | |
void | mdb_cursor_close (MDB_cursor *cursor) |
Close a cursor handle. | |
int | mdb_cursor_renew (MDB_txn *txn, MDB_cursor *cursor) |
Renew a cursor handle. | |
MDB_txn * | mdb_cursor_txn (MDB_cursor *cursor) |
Return the cursor’s transaction handle. | |
MDB_dbi | mdb_cursor_dbi (MDB_cursor *cursor) |
Return the cursor’s database handle. | |
int | mdb_cursor_get (MDB_cursor *cursor, MDB_val *key, MDB_val *data, MDB_cursor_op op) |
Retrieve by cursor. | |
int | mdb_cursor_put (MDB_cursor *cursor, MDB_val *key, MDB_val *data, unsigned int flags) |
Store by cursor. | |
int | mdb_cursor_del (MDB_cursor *cursor, unsigned int flags) |
Delete current key/data pair. | |
int | mdb_cursor_count (MDB_cursor *cursor, size_t *countp) |
Return count of duplicates for current key. | |
int | mdb_cmp (MDB_txn *txn, MDB_dbi dbi, const MDB_val *a, const MDB_val *b) |
Compare two data items according to a particular database. | |
int | mdb_dcmp (MDB_txn *txn, MDB_dbi dbi, const MDB_val *a, const MDB_val *b) |
Compare two data items according to a particular database. | |
others
Functions | |
---|---|
int | mdb_reader_list (MDB_env *env, MDB_msg_func *func, void *ctx) |
Dump the entries in the reader lock table. | |
int | mdb_reader_check (MDB_env *env, int *dead) |
Check for stale entries in the reader lock table. |
Modules(一些硬编码的值,比如flags)
Modules | |
---|---|
Version Macros | |
Environment Flags | |
Database Flags | |
Write Flags | |
Copy Flags | |
Return Codes | |
Data Structures
Data Structures | |
---|---|
struct | MDB_val |
Generic structure used for passing keys and data in and out of the database. More… | |
struct | MDB_stat |
Statistics for a database in the environment. More… | |
struct | MDB_envinfo |
Information about the environment. More… | |
Macros
Macros | |
---|---|
#define | mdb_open(txn, name, flags, dbi) mdb_dbi_open(txn,name,flags,dbi) |
#define | mdb_close(env, dbi) mdb_dbi_close(env,dbi) |
Typedefs
Typedefs | |
---|---|
typedef unsigned int | MDB_dbi |
A handle for an individual database in the DB environment. | |
typedef int( | MDB_cmp_func )(const MDB_val *a, const MDB_val *b) |
A callback function used to compare two keys in a database. | |
typedef void( | MDB_rel_func )(MDB_val *item, void *oldptr, void *newptr, void *relctx) |
A callback function used to relocate a position-dependent data item in a fixed-address database. | |
typedef void | MDB_assert_func (MDB_env *env, const char *msg) |
A callback function for most LMDB assert() failures, called before printing the message and aborting. | |
typedef int( | MDB_msg_func )(const char *msg, void *ctx) |
A callback function used to print a message from the library. | |
Enumerations
Enumerations | |
---|---|
enum | MDB_cursor_op { MDB_FIRST, MDB_FIRST_DUP, MDB_GET_BOTH, MDB_GET_BOTH_RANGE, MDB_GET_CURRENT, MDB_GET_MULTIPLE, MDB_LAST, MDB_LAST_DUP, MDB_NEXT, MDB_NEXT_DUP, MDB_NEXT_MULTIPLE, MDB_NEXT_NODUP, MDB_PREV, MDB_PREV_DUP, MDB_PREV_NODUP, MDB_SET, MDB_SET_KEY, MDB_SET_RANGE } |
Cursor Get operations. More… | |
Caveats
haven’t read yet