Gitingest

Include files under: 50kB

Private Repository NEW

Summary

Repository: shawnw/useful_sqlite_extensions
Files analyzed: 38

Estimated tokens: 138.5k

Download

Directory Structure

Directory structure:

└── shawnw-useful_sqlite_extensions/

    ├── README.md

    ├── LICENSE

    ├── docs/

    │   ├── blob_funcs.html

    │   ├── blob_funcs.md

    │   ├── bloom_filter1.html

    │   ├── bloom_filter1.md

    │   ├── json_funcs.html

    │   ├── json_funcs.md

    │   ├── Makefile

    │   ├── math_funcs.html

    │   ├── math_funcs.md

    │   ├── pcre2_funcs.html

    │   ├── pcre2_funcs.md

    │   ├── posix_re_funcs.html

    │   ├── posix_re_funcs.md

    │   ├── string_funcs.html

    │   └── string_funcs.md

    ├── src/

    │   ├── blob_funcs.c

    │   ├── bloom_filter.c

    │   ├── cJSON.c

    │   ├── cJSON.h

    │   ├── CMakeLists.txt

    │   ├── config.h.in

    │   ├── egc_str_funcs.c

    │   ├── group.c

    │   ├── icu_extras.c

    │   ├── json_funcs.c

    │   ├── math_funcs.c

    │   ├── more_str_funcs.c

    │   ├── passwd.c

    │   ├── pcre2_funcs.c

    │   ├── posix_re_funcs.c

    │   ├── stripaccents.c

    │   └── cmake/

    │       └── FindPCRE2.cmake

    └── tools/

        ├── README.md

        ├── csv2sqlite

        ├── demo_csvimport.sh

        └── table2sql

Files Content

(Files content cropped to 300k characters, download full ingest to see more)
================================================
FILE: README.md
================================================
Useful (?) Sqlite3 Stuff
========================

Extensions
----------

A handful of hopefully useful Sqlite3 extensions. Written mostly as a
way to get more familiar with the extension frameworks. See the files
in the *docs/* directory for details on individual extensions.

* **string_funcs** - Lots of extra Unicode-aware text handling functions
  and more. Includes all of the standard ICU module as well.
* **math_funcs** - Assorted floating-point functions.
* **blob_funcs** - Assorted functions that act on blobs.
* **pcre2_funcs** - Regular expressions using PCRE2
* **posix_re_funcs** - Regular expressions using POSIX Extended and Basic syntax.
* **json_funcs** - Extra JSON functions.
* **bloom_filter1** - Bloom filter indexes.

### Build Instructions ###

    % mkdir build
    % cd build
    % cmake -DCMAKE_BUILD_TYPE=RelWithDebInfo ../src
    % make
    % sqlite3
    sqlite3> .load ./libstring_funcs
    sqlite3> SELECT regexp_substr('abc def ghi', '[a-z]{3}', 1, 2);
    etc.

cmake will only build extensions that satisfy dependencies (No ICU dev
package installed, no libstring_funcs, for example).

A C99 compiler and standard library is required.

Tools
-----

Assorted scripts in the `tools/` directory. See the
[README](tools/README.md) there for details.

* **csv2sqlite** - Import CSV files into SQLite databases.
* **table2sql** - Convert ASCII art tables to SQL statements.

To-Do
-----

* Some of the string functions do a lot of reallocation and could
  stand to be improved.
* Test cases!
* Add more stuff.

License
-------

MIT.



================================================
FILE: LICENSE
================================================
Copyright 2019 Shawn Wagner <shawnw.mobile@gmail.com>

Permission is hereby granted, free of charge, to any person obtaining
a copy of this software and associated documentation files (the
"Software"), to deal in the Software without restriction, including
without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject to
the following conditions:

The above copyright notice and this permission notice shall be
included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.


================================================
FILE: docs/blob_funcs.html
================================================
<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml" lang="" xml:lang="">
<head>
  <meta charset="utf-8" />
  <meta name="generator" content="pandoc" />
  <meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes" />
  <title>Extra blob functions</title>
  <style type="text/css">
      code{white-space: pre-wrap;}
      span.smallcaps{font-variant: small-caps;}
      span.underline{text-decoration: underline;}
      div.column{display: inline-block; vertical-align: top; width: 50%;}
  </style>
  <!--[if lt IE 9]>
    <script src="//cdnjs.cloudflare.com/ajax/libs/html5shiv/3.7.3/html5shiv-printshiv.min.js"></script>
  <![endif]-->
</head>
<body>
<header>
<h1 class="title">Extra blob functions</h1>
</header>
<nav id="TOC">
<ul>
<li><a href="#introduction">Introduction</a></li>
<li><a href="#scalar-functions">Scalar Functions</a><ul>
<li><a href="#formatting">Formatting</a></li>
<li><a href="#message-digests">Message Digests</a></li>
<li><a href="#encryption">Encryption</a></li>
<li><a href="#compression">Compression</a></li>
<li><a href="#uuids">UUIDs</a></li>
</ul></li>
<li><a href="#aggregate-functions">Aggregate Functions</a></li>
</ul>
</nav>
<h1 id="introduction">Introduction</h1>
<p>This Sqlite3 extension module adds assorted functions for working with BLOB types (And strings). Requires OpenSSL.</p>
<p>Largely influenced by MySQL functions in features and names.</p>
<h1 id="scalar-functions">Scalar Functions</h1>
<h2 id="formatting">Formatting</h2>
<h3 id="unhex">UNHEX()</h3>
<ul>
<li>UNHEX(str)</li>
</ul>
<p>Does the opposite of the built in <code>HEX()</code> function - given a string that’s Base16 encoded, returns the decoded BLOB. Returns <code>NULL</code> if the string is not Base16 encoded or isn’t an even length, or if passed a <code>NULL</code>.</p>
<h3 id="to_base64">TO_BASE64()</h3>
<ul>
<li>TO_BASE64(blob)</li>
</ul>
<p>Returns <code>blob</code> encoded as a Base64 string. To match MySQL, the result is broken up into multiple lines if long enough.</p>
<h3 id="from_base64">FROM_BASE64()</h3>
<ul>
<li>FROM_BASE64(str)</li>
</ul>
<p>Returns <code>str</code> decoded from Base64 into a BLOB. If the argument is <code>NULL</code> or an invalid Base64 string, returns <code>NULL</code>.</p>
<h2 id="message-digests">Message Digests</h2>
<p>If any of these functions are passed a <code>NULL</code> argument, return <code>NULL</code>.</p>
<h3 id="md5">MD5()</h3>
<ul>
<li>MD5(b)</li>
</ul>
<p>Returns the MD5 digest of its blob argument as a Base16 encoded string.</p>
<h3 id="sha1">SHA1()</h3>
<ul>
<li>SHA1(b)</li>
</ul>
<p>Returns the SHA1 digest of its blob argument as a Base16 encoded string.</p>
<h3 id="sha2">SHA2()</h3>
<ul>
<li>SHA2(b)</li>
<li>SHA2(b, i)</li>
</ul>
<p>Returns the <code>i</code>-bit SHA2 digest of its blob argument as a Base16 encoded string. <code>i</code> can be 224, 256, 384, 512, or 0 (Which is treated as 256). Defaults to 256 if ommitted.</p>
<h3 id="sha3">SHA3()</h3>
<ul>
<li>SHA3(b)</li>
<li>SHA3(b, i)</li>
</ul>
<p>Returns the <code>i</code>-bit SHA3 digest of its blob argument as a Base16 encoded string. <code>i</code> can be 224, 256, 384, or 512. Defaults to 256 if omitted.</p>
<p>Implementation taken from the <code>shathree</code> extension in the sqlite3 source tree. This version returns a Base16 string, not blob, though.</p>
<h3 id="create_digest">CREATE_DIGEST()</h3>
<ul>
<li>CREATE_DIGEST(algo, b)</li>
</ul>
<p>Returns the <code>algo</code> digest of <code>b</code> as a blob. <code>algo</code> can be any message digest supported by OpenSSL, including but not limited to, ‘md5’, ‘rmd160’, ‘sha1’, ‘sha256’, ‘sha512’, etc.</p>
<p>You can get the complete list with <code>openssl list --digest-commands</code> at a shell.</p>
<h3 id="hmac">HMAC()</h3>
<ul>
<li>HMAC(algo, secret, b)</li>
</ul>
<p>Returns the HMAC of <code>b</code>, using the <code>algo</code> message digest algorithm, and secret <code>secret</code> as a Base16 encoded string.</p>
<h3 id="crc32">CRC32()</h3>
<ul>
<li>CRC32(b)</li>
</ul>
<p>Computes the <a href="https://en.wikipedia.org/wiki/Cyclic_redundancy_check">CRC-32</a> checksum of its blob argument and returns the result as an integer.</p>
<h3 id="crc32c">CRC32C()</h3>
<ul>
<li>CRC32C(b)</li>
</ul>
<p>Computes the CRC-32C checksum of its blob argument and returns the result as an integer. Currently only available on x86 processors.</p>
<h2 id="encryption">Encryption</h2>
<h3 id="aes_encrypt">AES_ENCRYPT()</h3>
<ul>
<li>AES_ENCRYPT(str, key)</li>
</ul>
<p>Returns a BLOB of <code>str</code> encrypted using the <strong>AES-128-ECB</strong> algorithm with key <code>key</code>. The key should be a 128 bit (16 byte) or larger blob or string.</p>
<h3 id="aes_decrypt">AES_DECRYPT()</h3>
<ul>
<li>AES_DECRYPT(aes, key)</li>
</ul>
<p>Returns the decrypted <code>aes</code>, which is a BLOB holding padded, <strong>AES-128-ECB</strong> encrypted data.</p>
<h2 id="compression">Compression</h2>
<p>These functions require the zlib library and work with arbitrary blobs of data.</p>
<p>For compressing non-ASCII text, consider the Unicode-specific compression functions in the <code>string_funcs</code> module.</p>
<h3 id="compress">COMPRESS()</h3>
<ul>
<li>COMPRESS(b)</li>
</ul>
<p>Returns its blob argument as a zlib-compressed blob. The first four bytes are the length of the uncompressed text, in big-endian format. Note: This makes it incompatible with the <a href="https://www3.sqlite.org/cgi/src/artifact/dd4f8a6d0baccff3">compress module</a> in the sqlite source tree, which uses a variable-width encoding.</p>
<h3 id="uncompress">UNCOMPRESS()</h3>
<ul>
<li>UNCOMPRESS(b)</li>
</ul>
<p>Returns its decompressed argument as a blob. <code>b</code> must be a blob that was compressed with <code>COMPRESS()</code>.</p>
<h2 id="uuids">UUIDs</h2>
<h3 id="uuid">UUID()</h3>
<ul>
<li>UUID()</li>
</ul>
<p>Generate a new type 4 (Random) UUID and return it as a blob.</p>
<h3 id="bin_to_uuid">BIN_TO_UUID()</h3>
<ul>
<li>BIN_TO_UUID(b)</li>
</ul>
<p>Convert a UUID blob to a string representation.</p>
<h3 id="uuid_to_bin">UUID_TO_BIN()</h3>
<ul>
<li>UUID_TO_BIN(s)</li>
</ul>
<p>Convert a UUID string to a blob representation.</p>
<h3 id="is_uuid">IS_UUID()</h3>
<ul>
<li>IS_UUID(s/b)</li>
</ul>
<p>Returns 1 if <code>s</code> is a valid string or blob representation of a UUID, 0 if not, and NULL if NULL.</p>
<h1 id="aggregate-functions">Aggregate Functions</h1>
<p>TODO: Aggregate versions of the digest functions?</p>
</body>
</html>



================================================
FILE: docs/blob_funcs.md
================================================
% Extra blob functions

Introduction
============

This Sqlite3 extension module adds assorted functions for working with
BLOB types (And strings). Requires OpenSSL.

Largely influenced by MySQL functions in features and names.

Scalar Functions
================

Formatting
----------

### UNHEX()

* UNHEX(str)

Does the opposite of the built in `HEX()` function - given a string
that's Base16 encoded, returns the decoded BLOB. Returns `NULL` if
the string is not Base16 encoded or isn't an even length, or if
passed a `NULL`.

### TO_BASE64()

* TO_BASE64(blob)

Returns `blob` encoded as a Base64 string. To match MySQL, the result
is broken up into multiple lines if long enough.

### FROM_BASE64()

* FROM_BASE64(str)

Returns `str` decoded from Base64 into a BLOB. If the argument is
`NULL` or an invalid Base64 string, returns `NULL`.

Message Digests
---------------

If any of these functions are passed a `NULL` argument, return `NULL`.

### MD5() ###

* MD5(b)

Returns the MD5 digest of its blob argument as a Base16 encoded string.

### SHA1() ####

* SHA1(b)

Returns the SHA1 digest of its blob argument as a Base16 encoded string.

### SHA2() ####

* SHA2(b)
* SHA2(b, i)

Returns the `i`-bit SHA2 digest of its blob argument as a Base16
encoded string. `i` can be 224, 256, 384, 512, or 0 (Which is treated
as 256). Defaults to 256 if ommitted.

### SHA3() ###

* SHA3(b)
* SHA3(b, i)

Returns the `i`-bit SHA3 digest of its blob argument as a Base16
encoded string.  `i` can be 224, 256, 384, or 512. Defaults to 256 if
omitted.

Implementation taken from the `shathree` extension in the sqlite3
source tree. This version returns a Base16 string, not blob, though.

### CREATE_DIGEST()

* CREATE_DIGEST(algo, b)

Returns the `algo` digest of `b` as a blob. `algo` can be any message
digest supported by OpenSSL, including but not limited to, 'md5', 'rmd160',
'sha1', 'sha256', 'sha512', etc.

You can get the complete list with `openssl list --digest-commands` at
a shell.

### HMAC()

* HMAC(algo, secret, b)

Returns the HMAC of `b`, using the `algo` message digest algorithm,
and secret `secret` as a Base16 encoded string.

### CRC32()

* CRC32(b)

Computes the [CRC-32] checksum of its blob argument and returns the
result as an integer.

[CRC-32]: https://en.wikipedia.org/wiki/Cyclic_redundancy_check

### CRC32C()

* CRC32C(b)

Computes the CRC-32C checksum of its blob argument and returns the
result as an integer. Currently only available on x86 processors.

Encryption
----------

### AES_ENCRYPT()

* AES_ENCRYPT(str, key)

Returns a BLOB of `str` encrypted using the **AES-128-ECB** algorithm
with key `key`. The key should be a 128 bit (16 byte) or larger blob
or string.

### AES_DECRYPT()

* AES_DECRYPT(aes, key)

Returns the decrypted `aes`, which is a BLOB holding padded,
**AES-128-ECB** encrypted data.

Compression
-----------

These functions require the zlib library and work with arbitrary blobs
of data.

For compressing non-ASCII text, consider the Unicode-specific
compression functions in the `string_funcs` module.

### COMPRESS()

* COMPRESS(b)

Returns its blob argument as a zlib-compressed blob. The first four
bytes are the length of the uncompressed text, in big-endian
format. Note: This makes it incompatible with the [compress module] in
the sqlite source tree, which uses a variable-width encoding.

[compress module]: https://www3.sqlite.org/cgi/src/artifact/dd4f8a6d0baccff3

### UNCOMPRESS()

* UNCOMPRESS(b)

Returns its decompressed argument as a blob. `b` must be a blob that
was compressed with `COMPRESS()`.

UUIDs
-----

### UUID()

* UUID()

Generate a new type 4 (Random) UUID and return it as a blob.

### BIN_TO_UUID()

* BIN\_TO\_UUID(b)

Convert a UUID blob to a string representation.

### UUID_TO_BIN()

* UUID\_TO\_BIN(s)

Convert a UUID string to a blob representation.

### IS_UUID()

* IS_UUID(s/b)

Returns 1 if `s` is a valid string or blob representation of a UUID, 0 if not,
and NULL if NULL.

Aggregate Functions
===================

TODO: Aggregate versions of the digest functions?



================================================
FILE: docs/bloom_filter1.html
================================================
<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml" lang="" xml:lang="">
<head>
  <meta charset="utf-8" />
  <meta name="generator" content="pandoc" />
  <meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes" />
  <title>Bloom Filter Virtual Table</title>
  <style type="text/css">
      code{white-space: pre-wrap;}
      span.smallcaps{font-variant: small-caps;}
      span.underline{text-decoration: underline;}
      div.column{display: inline-block; vertical-align: top; width: 50%;}
  </style>
  <!--[if lt IE 9]>
    <script src="//cdnjs.cloudflare.com/ajax/libs/html5shiv/3.7.3/html5shiv-printshiv.min.js"></script>
  <![endif]-->
</head>
<body>
<header>
<h1 class="title">Bloom Filter Virtual Table</h1>
</header>
<nav id="TOC">
<ul>
<li><a href="#introduction">Introduction</a></li>
<li><a href="#usage">Usage</a><ul>
<li><a href="#creating-a-table">Creating a table</a></li>
<li><a href="#populating-the-table">Populating the table</a></li>
<li><a href="#querying-the-table">Querying the table</a></li>
</ul></li>
<li><a href="#example">Example</a></li>
<li><a href="#implementation-notes">Implementation notes</a></li>
</ul>
</nav>
<h1 id="introduction">Introduction</h1>
<p><a href="https://en.wikipedia.org/wiki/Bloom_filter">Bloom filters</a> are a classic data structure for telling, in a fast and compact manner, if a given value has a chance of being present in a set. This extension module provides virtual tables that act as an interface to a bloom filter. They can be useful as a fast index to tell if a value is probably in a table or certainly isn’t.</p>
<h1 id="usage">Usage</h1>
<h2 id="creating-a-table">Creating a table</h2>
<pre><code>CREATE VIRTUAL TABLE foo USING bloom_filter1(n, p, k);</code></pre>
<p>The constructor takes three arguments:</p>
<ul>
<li><code>n</code> – the expected number of elements that will be stored in the filter.</li>
<li><code>p</code> – the probability of false positives, as a number between 0 and 1.0. The higher the number, the higher the chance of a false positive.</li>
<li><code>k</code> – the number of times to hash each value into the filter.</li>
</ul>
<p>The only mandatory one is <code>n</code>. If omitted, <code>p</code> defaults to <strong>0.01</strong> (1%), and an ideal value of <code>k</code> is calculated.</p>
<h2 id="populating-the-table">Populating the table</h2>
<pre><code>INSERT INTO foo VALUES (&#39;key 1&#39;), (&#39;key 2&#39;), ...;</code></pre>
<p>All values are first converted to <code>BLOB</code>s and the resuling bytes are hashed. This means that integers and floating point numbers can produce funny results. If you insert <code>1.0</code>, looking for <code>1</code> won’t find it, but looking for <code>'1.0'</code> will.</p>
<h2 id="querying-the-table">Querying the table</h2>
<pre><code>SELECT * from foo(&#39;key 1&#39;);</code></pre>
<p>Returns a single row with a single column set to <strong>1</strong> if the key is found in the filter. No rows are returned if the key is not present. This can be used with <code>EXISTS</code> and <code>IN</code> in queries.</p>
<p>I’ve thought about making <code>MATCH</code> work too, but it has some issues - you can’t have multiple modules loaded at once that use it, and syntax would be ugly in its own way - you’d have to provide the table name as a string.</p>
<h1 id="example">Example</h1>
<pre><code>sqlite&gt; CREATE TABLE plants(name);
sqlite&gt; CREATE VIRTUAL TABLE vegetables USING bloom_filter1(20);
sqlite&gt; INSERT INTO plants VALUES (&#39;apple&#39;), (&#39;asparagus&#39;), (&#39;cabbage&#39;), (&#39;grass&#39;);
sqlite&gt; INSERT INTO vegetables VALUES (&#39;asparagus&#39;), (&#39;cabbage&#39;);
sqlite&gt; SELECT * FROM plants WHERE EXISTS (SELECT * FROM vegetables(name));
asparagus
cabbage
sqlite3&gt; SELECT * FROM plants WHERE 1 NOT IN vegetables(name);
apple
grass</code></pre>
<h1 id="implementation-notes">Implementation notes</h1>
<p>The underlying hash function is <a href="https://en.wikipedia.org/wiki/SipHash">SipHash</a>.</p>
<p>For each virtual table <code>foo</code>, a backing table <code>foo_storage</code> is created, with one row. It has a blob holding the bloom filter, and some informational columns:</p>
<ul>
<li><code>n</code> – how many elements the table is intended for. The actual number of elements inserted into it is not tracked.</li>
<li><code>m</code> – the number of bits used in the filter.</li>
<li><code>p</code> – the false positive chance.</li>
<li><code>k</code> – the number of hash functions used.</li>
</ul>
<p>The Sqlite3 incremental blob API is used to avoid reading and writing large amounts of data at once.</p>
</body>
</html>



================================================
FILE: docs/bloom_filter1.md
================================================
% Bloom Filter Virtual Table

Introduction
============

[Bloom filters] are a classic data structure for telling, in a fast
and compact manner, if a given value has a chance of being present in
a set. This extension module provides virtual tables that act as an
interface to a bloom filter. They can be useful as a fast index to
tell if a value is probably in a table or certainly isn't.

[Bloom filters]: https://en.wikipedia.org/wiki/Bloom_filter

Usage
=====

Creating a table
----------------

    CREATE VIRTUAL TABLE foo USING bloom_filter1(n, p, k);
    
The constructor takes three arguments:

* `n` -- the expected number of elements that will be stored in the filter.
* `p` -- the probability of false positives, as a number between 0 and
  1.0. The higher the number, the higher the chance of a false
  positive.
* `k` -- the number of times to hash each value into the filter.

The only mandatory one is `n`. If omitted, `p` defaults to **0.01**
(1%), and an ideal value of `k` is calculated.

Populating the table
--------------------

    INSERT INTO foo VALUES ('key 1'), ('key 2'), ...;

All values are first converted to `BLOB`s and the resuling bytes are
hashed. This means that integers and floating point numbers can
produce funny results. If you insert `1.0`, looking for `1` won't find
it, but looking for `'1.0'` will.

Querying the table
------------------

    SELECT * from foo('key 1');

Returns a single row with a single column set to **1** if the key is
found in the filter. No rows are returned if the key is not
present. This can be used with `EXISTS` and `IN` in queries.

I've thought about making `MATCH` work too, but it has some issues -
you can't have multiple modules loaded at once that use it, and syntax
would be ugly in its own way - you'd have to provide the table name as
a string.

Example
=======

    sqlite> CREATE TABLE plants(name);
    sqlite> CREATE VIRTUAL TABLE vegetables USING bloom_filter1(20);
    sqlite> INSERT INTO plants VALUES ('apple'), ('asparagus'), ('cabbage'), ('grass');
    sqlite> INSERT INTO vegetables VALUES ('asparagus'), ('cabbage');
    sqlite> SELECT * FROM plants WHERE EXISTS (SELECT * FROM vegetables(name));
    asparagus
    cabbage
    sqlite3> SELECT * FROM plants WHERE 1 NOT IN vegetables(name);
    apple
    grass
    
Implementation notes
====================

The underlying hash function is [SipHash].

For each virtual table `foo`, a backing table `foo_storage` is
created, with one row. It has a blob holding the bloom filter, and
some informational columns:

* `n` -- how many elements the table is intended for. The actual number
  of elements inserted into it is not tracked.
* `m` -- the number of bits used in the filter.
* `p` -- the false positive chance.
* `k` -- the number of hash functions used.

The Sqlite3 incremental blob API is used to avoid reading and writing
large amounts of data at once.

[SipHash]: https://en.wikipedia.org/wiki/SipHash



================================================
FILE: docs/json_funcs.html
================================================
<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml" lang="" xml:lang="">
<head>
  <meta charset="utf-8" />
  <meta name="generator" content="pandoc" />
  <meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes" />
  <title>Extra JSON functions</title>
  <style type="text/css">
      code{white-space: pre-wrap;}
      span.smallcaps{font-variant: small-caps;}
      span.underline{text-decoration: underline;}
      div.column{display: inline-block; vertical-align: top; width: 50%;}
  </style>
  <!--[if lt IE 9]>
    <script src="//cdnjs.cloudflare.com/ajax/libs/html5shiv/3.7.3/html5shiv-printshiv.min.js"></script>
  <![endif]-->
</head>
<body>
<header>
<h1 class="title">Extra JSON functions</h1>
</header>
<nav id="TOC">
<ul>
<li><a href="#introduction">Introduction</a></li>
<li><a href="#functions">Functions</a></li>
</ul>
</nav>
<h1 id="introduction">Introduction</h1>
<p>This Sqlite3 extension adds some extra JSON handling functions to supplement the standard <a href="https://www.sqlite.org/json1.html">JSON1</a> extension. They’re mostly inspired by <a href="https://dev.mysql.com/doc/refman/8.0/en/json-function-reference.html">MySQL JSON functions</a>.</p>
<h1 id="functions">Functions</h1>
<h3 id="json_equal">JSON_EQUAL()</h3>
<ul>
<li>JSON_EQUAL(json, json)</li>
</ul>
<p>Returns 1 if the two JSON values are equivalent, 0 if not.</p>
<h3 id="json_length">JSON_LENGTH()</h3>
<ul>
<li>JSON_LENGTH(json)</li>
<li>JSON_LENGTH(json, path)</li>
</ul>
<p>If <code>json</code> (Either the entire object, or the part of it at <code>path</code>) is an object or array, return the number of elements in it. Otherwise, returns 1.</p>
<h3 id="json_pretty">JSON_PRETTY()</h3>
<ul>
<li>JSON_PRETTY(json)</li>
</ul>
<p>Returns <code>json</code>, pretty-printed for human readability. The exact format depends on the version of sqlite3 that the extension was compiled against; 3.24 and newer produce MySQL style output, older produces <a href="https://github.com/DaveGamble/cJSON">cJSON</a> style.</p>
<h3 id="json_keys">JSON_KEYS()</h3>
<ul>
<li>JSON_KEYS(json)</li>
<li>JSON_KEYS(json, path)</li>
</ul>
<p>Returns a JSON array of the keys of the given JSON object.</p>
<p>*** JSON_CONTAINS_PATH()</p>
<ul>
<li>JSON_CONTAINS_PATH(json, path)</li>
</ul>
<p>Returns 1 if the json object has data at the given path, 0 otherwise.</p>
</body>
</html>



================================================
FILE: docs/json_funcs.md
================================================
% Extra JSON functions

Introduction
============

This Sqlite3 extension adds some extra JSON handling functions to
supplement the standard [JSON1] extension. They're mostly inspired by
[MySQL JSON functions].

[JSON1]: https://www.sqlite.org/json1.html
[MySQL JSON functions]: https://dev.mysql.com/doc/refman/8.0/en/json-function-reference.html

Functions
=========

### JSON_EQUAL()

* JSON_EQUAL(json, json)

Returns 1 if the two JSON values are equivalent, 0 if not.

### JSON_LENGTH()

* JSON_LENGTH(json)
* JSON_LENGTH(json, path)

If `json` (Either the entire object, or the part of it at `path`) is
an object or array, return the number of elements in it. Otherwise,
returns 1.

### JSON_PRETTY()

* JSON_PRETTY(json)

Returns `json`, pretty-printed for human readability. The exact format
depends on the version of sqlite3 that the extension was compiled
against; 3.24 and newer produce MySQL style output, older produces
[cJSON] style.

[cJSON]: https://github.com/DaveGamble/cJSON

### JSON_KEYS()

* JSON_KEYS(json)
* JSON_KEYS(json, path)

Returns a JSON array of the keys of the given JSON object.

*** JSON_CONTAINS_PATH()

* JSON_CONTAINS_PATH(json, path)

Returns 1 if the json object has data at the given path, 0 otherwise.



================================================
FILE: docs/Makefile
================================================
.PHONY: all

all: string_funcs.html math_funcs.html bloom_filter1.html \
	blob_funcs.html pcre2_funcs.html posix_re_funcs.html json_funcs.html

%.html:  %.md
	pandoc -f markdown -t html5 --mathjax -s --toc --toc-depth=2 -o $@ $<



================================================
FILE: docs/math_funcs.html
================================================
<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml" lang="" xml:lang="">
<head>
  <meta charset="utf-8" />
  <meta name="generator" content="pandoc" />
  <meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes" />
  <title>Math and Numeric Functions</title>
  <style type="text/css">
      code{white-space: pre-wrap;}
      span.smallcaps{font-variant: small-caps;}
      span.underline{text-decoration: underline;}
      div.column{display: inline-block; vertical-align: top; width: 50%;}
  </style>
  <script src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/MathJax.js?config=TeX-AMS_CHTML-full" type="text/javascript"></script>
  <!--[if lt IE 9]>
    <script src="//cdnjs.cloudflare.com/ajax/libs/html5shiv/3.7.3/html5shiv-printshiv.min.js"></script>
  <![endif]-->
</head>
<body>
<header>
<h1 class="title">Math and Numeric Functions</h1>
</header>
<nav id="TOC">
<ul>
<li><a href="#introduction">Introduction</a></li>
<li><a href="#scalar-functions">Scalar Functions</a><ul>
<li><a href="#trigonometric">Trigonometric</a></li>
<li><a href="#roots-powers-and-logs">Roots, Powers and Logs</a></li>
<li><a href="#rounding">Rounding</a></li>
<li><a href="#math">Math</a></li>
<li><a href="#other">Other</a></li>
</ul></li>
<li><a href="#aggregate-functions">Aggregate Functions</a><ul>
<li><a href="#math-1">Math</a></li>
<li><a href="#bitwise">Bitwise</a></li>
<li><a href="#statistics">Statistics</a></li>
</ul></li>
</ul>
</nav>
<h1 id="introduction">Introduction</h1>
<p>Mostly lifted from MySQL and Postgres. Unless otherwise documented, they do the same thing as the C function of the same name. Pass them <code>NULL</code>, get <code>NULL</code> back. Do something that would make them return a <code>NaN</code>, get <code>NULL</code> back. In the arguments, <em>d</em> means a floating point number, <em>i</em> means an integer. <em>d/i</em> means either.</p>
<h1 id="scalar-functions">Scalar Functions</h1>
<h2 id="trigonometric">Trigonometric</h2>
<ul>
<li>ACOS(d)</li>
<li>ACOSH(d)</li>
<li>ASIN(d)</li>
<li>ASINH(d)</li>
<li>ATAN(d)</li>
<li>ATAN2(d,d)</li>
<li>ATANH(d)</li>
<li>COS(d)</li>
<li>COSH(d)</li>
<li>COT(d) – Cotangent</li>
<li>SIN(d)</li>
<li>SINH(d)</li>
<li>TAN(d)</li>
<li>TANH(d)</li>
<li>DEGREES(d) – Radians to degrees</li>
<li>RADIANS(d) – Degrees to radians</li>
</ul>
<h2 id="roots-powers-and-logs">Roots, Powers and Logs</h2>
<ul>
<li>CBRT(d) – <span class="math inline">\(\sqrt[3]{x}\)</span></li>
<li>EXP(d) – <span class="math inline">\(e^{x}\)</span></li>
<li>EXP2(d) – <span class="math inline">\(2^{x}\)</span></li>
<li>EXPM1(d) – <span class="math inline">\(e^{x-1}\)</span></li>
<li>HYPOT(d, d) – <span class="math inline">\(\sqrt{x^{2} + y^{2}}\)</span></li>
<li>LN(d) – <span class="math inline">\(\log_{e} x\)</span></li>
<li>LOG(d) – Base ℯ like MySQL, not base 10 like Postgresql.</li>
<li>LOG(d,d) – <span class="math inline">\(\log_{x} y\)</span></li>
<li>LOG1P(d) – <span class="math inline">\(\log_{e} (x+1)\)</span></li>
<li>LOG10(d) – <span class="math inline">\(\log_{10} x\)</span></li>
<li>LOG2(d) – <span class="math inline">\(\log_{2} x\)</span></li>
<li>POWER(d,d) – <span class="math inline">\(x^{y}\)</span></li>
<li>SQRT(d) – <span class="math inline">\(\sqrt{x}\)</span></li>
</ul>
<h2 id="rounding">Rounding</h2>
<ul>
<li>CEIL(d)</li>
<li>FLOOR(d)</li>
<li>ROUND(d) – Overrides the standard one-argument <code>ROUND()</code>. Rounds halfway cases away from zero.</li>
<li>TRUNC(d) – Rounds towards zero.</li>
</ul>
<h2 id="math">Math</h2>
<ul>
<li>DIV(i, i) – Integer division</li>
<li>MOD(i, i) – Integer remainder</li>
<li>SIGN(d/i)</li>
<li>PI() – π</li>
</ul>
<h2 id="other">Other</h2>
<ul>
<li>BIT_COUNT(i) – Returns the number of set bits in its integer argument.</li>
</ul>
<h1 id="aggregate-functions">Aggregate Functions</h1>
<h2 id="math-1">Math</h2>
<ul>
<li>PRODUCT(d) – compute the product of a group. Can be used as a window function.</li>
</ul>
<h2 id="bitwise">Bitwise</h2>
<ul>
<li>BIT_OR(i)</li>
<li>BIT_XOR(i) – can be used as a window function</li>
<li>BIT_AND(i)</li>
</ul>
<h2 id="statistics">Statistics</h2>
<p>All of these can be used as window functions.</p>
<ul>
<li>CORR(d, d) – Correlation coefficient.</li>
<li>COVAR_POP(d, d) – Population covariance.</li>
<li>COVAR_SAMP(d, d) – Sample covariance.</li>
<li>STDDEV_POP(d) – Population standard deviation.</li>
<li>STDDEV_SAMP(d) – Sample standard deviation.</li>
<li>VAR_POP(d) – Population variance.</li>
<li>VAR_SAMP(d) – Sample variance.</li>
<li>GEO_MEAN(d) – Geometric mean.</li>
<li>HARM_MEAN(d) – Harmonic mean.</li>
<li>MEDIAN(d)</li>
<li>MODE(d)</li>
<li>Q1(d) – 1st Quartile Value.</li>
<li>Q3(d) – 3rd Quartile Value.</li>
<li>IQR(d) – Interquartile Range.</li>
</ul>
<h3 id="linear-regression">Linear Regression</h3>
<ul>
<li>REGR_SLOPE(d,d) – The slope of a line.</li>
<li>REGR_INTERCEPT(d, d) – The y-intercept of the regression line.</li>
<li>REGR_COUNT(d, d) – The number of pairs used in regression calculations.</li>
<li>REGR_R2(d, d) – The coefficient of determination for the regression.</li>
<li>REGR_AVGX(d, d) – The average of the second values of the pairs.</li>
<li>REGR_AVGY(d, d) – The average of the first values of the pairs.</li>
<li>REGR_SXX(d, d)</li>
<li>REGR_SYY(d, d)</li>
<li>REGR_SXY(d, d)</li>
</ul>
</body>
</html>



================================================
FILE: docs/math_funcs.md
================================================
% Math and Numeric Functions

Introduction
============

Mostly lifted from MySQL and Postgres. Unless otherwise documented,
they do the same thing as the C function of the same name. Pass them
`NULL`, get `NULL` back. Do something that would make them return a
`NaN`, get `NULL` back. In the arguments, *d* means a floating point
number, *i* means an integer. *d/i* means either.

Scalar Functions
================

Trigonometric
-------------

* ACOS(d)
* ACOSH(d)
* ASIN(d)
* ASINH(d)
* ATAN(d)
* ATAN2(d,d)
* ATANH(d)
* COS(d)
* COSH(d)
* COT(d) -- Cotangent
* SIN(d)
* SINH(d)
* TAN(d)
* TANH(d)
* DEGREES(d) -- Radians to degrees 
* RADIANS(d) -- Degrees to radians

Roots, Powers and Logs
----------------------

* CBRT(d) -- $\sqrt[3]{x}$
* EXP(d) -- $e^{x}$
* EXP2(d) -- $2^{x}$
* EXPM1(d) -- $e^{x-1}$
* HYPOT(d, d) -- $\sqrt{x^{2} + y^{2}}$
* LN(d) -- $\log_{e} x$
* LOG(d) -- Base ℯ like MySQL, not base 10 like Postgresql.
* LOG(d,d) -- $\log_{x} y$
* LOG1P(d) -- $\log_{e} (x+1)$
* LOG10(d) -- $\log_{10} x$
* LOG2(d) -- $\log_{2} x$
* POWER(d,d) -- $x^{y}$
* SQRT(d) -- $\sqrt{x}$

Rounding
--------

* CEIL(d)
* FLOOR(d)
* ROUND(d) -- Overrides the standard one-argument `ROUND()`. Rounds
  halfway cases away from zero.
* TRUNC(d) -- Rounds towards zero.

Math
----

* DIV(i, i) -- Integer division
* MOD(i, i) -- Integer remainder
* SIGN(d/i)
* PI() -- π

Other
-----

* BIT_COUNT(i) -- Returns the number of set bits in its integer argument.

Aggregate Functions
===================

Math
----

* PRODUCT(d) -- compute the product of a group. Can be used as a
  window function.

Bitwise
-------

* BIT_OR(i)
* BIT_XOR(i) -- can be used as a window function
* BIT_AND(i)

Statistics
----------

All of these can be used as window functions.

* CORR(d, d) -- Correlation coefficient.
* COVAR_POP(d, d) -- Population covariance. 
* COVAR_SAMP(d, d) -- Sample covariance. 
* STDDEV_POP(d) -- Population standard deviation.
* STDDEV_SAMP(d) -- Sample standard deviation.
* VAR_POP(d) -- Population variance.
* VAR_SAMP(d) -- Sample variance.
* GEO_MEAN(d) -- Geometric mean.
* HARM_MEAN(d) -- Harmonic mean.
* MEDIAN(d)
* MODE(d)
* Q1(d) -- 1st Quartile Value.
* Q3(d) -- 3rd Quartile Value.
* IQR(d) -- Interquartile Range.

### Linear Regression

* REGR_SLOPE(d,d) -- The slope of a line.
* REGR_INTERCEPT(d, d) -- The y-intercept of the regression line.
* REGR_COUNT(d, d) -- The number of pairs used in regression calculations.
* REGR_R2(d, d) -- The coefficient of determination for the regression.
* REGR_AVGX(d, d) -- The average of the second values of the pairs.
* REGR_AVGY(d, d) -- The average of the first values of the pairs.
* REGR_SXX(d, d)
* REGR_SYY(d, d)
* REGR_SXY(d, d)



================================================
FILE: docs/pcre2_funcs.html
================================================
<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml" lang="" xml:lang="">
<head>
  <meta charset="utf-8" />
  <meta name="generator" content="pandoc" />
  <meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes" />
  <title>PCRE2 regular expression functions</title>
  <style type="text/css">
      code{white-space: pre-wrap;}
      span.smallcaps{font-variant: small-caps;}
      span.underline{text-decoration: underline;}
      div.column{display: inline-block; vertical-align: top; width: 50%;}
  </style>
  <!--[if lt IE 9]>
    <script src="//cdnjs.cloudflare.com/ajax/libs/html5shiv/3.7.3/html5shiv-printshiv.min.js"></script>
  <![endif]-->
</head>
<body>
<header>
<h1 class="title">PCRE2 regular expression functions</h1>
</header>
<nav id="TOC">
<ul>
<li><a href="#introduction">Introduction</a></li>
<li><a href="#functions">Functions</a><ul>
<li><a href="#matching">Matching</a></li>
<li><a href="#informational">Informational</a></li>
</ul></li>
<li><a href="#to-do">To-Do</a></li>
</ul>
</nav>
<h1 id="introduction">Introduction</h1>
<p>This Sqlite3 extension module provides MySQL inspired regular expression functions using the <a href="https://www.pcre.org">PCRE2</a> engine. Every plain function is also present with <strong>PCRE_</strong> prepended to the name, so multiple modules that provide RE functions can coexist portably. (See the <code>string_funcs</code> module for versions using <a href="http://site.icu-project.org">ICU</a> regular expressions.) The default names are used by whichever module was loaded last.</p>
<h1 id="functions">Functions</h1>
<h2 id="matching">Matching</h2>
<h3 id="regexp">REGEXP()</h3>
<ul>
<li>REGEXP(re, string)</li>
<li>string REGEXP re</li>
</ul>
<p>Returns 1 if <code>string</code> matches against <code>re</code>. The entire string must match.</p>
<h2 id="informational">Informational</h2>
<h3 id="pcre_version">PCRE_VERSION()</h3>
<p>Returns the version of PCRE2 being used.</p>
<h3 id="pcre_unicode_version">PCRE_UNICODE_VERSION()</h3>
<p>Returns the version of Unicode that PCRE2 is using.</p>
<h1 id="to-do">To-Do</h1>
<ul>
<li>Add the rest of the functions.</li>
</ul>
</body>
</html>



================================================
FILE: docs/pcre2_funcs.md
================================================
% PCRE2 regular expression functions

Introduction
============

This Sqlite3 extension module provides MySQL inspired regular
expression functions using the [PCRE2] engine. Every plain function is
also present with **PCRE_** prepended to the name, so multiple modules
that provide RE functions can coexist portably. (See the
`string_funcs` module for versions using [ICU] regular expressions.)
The default names are used by whichever module was loaded last.

[PCRE2]: https://www.pcre.org
[ICU]: http://site.icu-project.org

Functions
=========

Matching
--------

### REGEXP()

* REGEXP(re, string)
* string REGEXP re

Returns 1 if `string` matches against `re`. The entire string must match.

Informational
-------------

### PCRE_VERSION()

Returns the version of PCRE2 being used.

### PCRE_UNICODE_VERSION()

Returns the version of Unicode that PCRE2 is using.

To-Do
=====

* Add the rest of the functions.



================================================
FILE: docs/posix_re_funcs.html
================================================
<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml" lang="" xml:lang="">
<head>
  <meta charset="utf-8" />
  <meta name="generator" content="pandoc" />
  <meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes" />
  <title>POSIX regular expression functions</title>
  <style type="text/css">
      code{white-space: pre-wrap;}
      span.smallcaps{font-variant: small-caps;}
      span.underline{text-decoration: underline;}
      div.column{display: inline-block; vertical-align: top; width: 50%;}
  </style>
  <!--[if lt IE 9]>
    <script src="//cdnjs.cloudflare.com/ajax/libs/html5shiv/3.7.3/html5shiv-printshiv.min.js"></script>
  <![endif]-->
</head>
<body>
<header>
<h1 class="title">POSIX regular expression functions</h1>
</header>
<nav id="TOC">
<ul>
<li><a href="#introduction">Introduction</a></li>
<li><a href="#functions">Functions</a><ul>
<li><a href="#matching">Matching</a></li>
</ul></li>
<li><a href="#to-do">To-Do</a></li>
</ul>
</nav>
<h1 id="introduction">Introduction</h1>
<p>This Sqlite3 extension module provides regular expression functions using POSIX Extended and Basic syntax.</p>
<p>See the <code>string_funcs</code> module for versions using <a href="http://site.icu-project.org">ICU</a> regular expressions and <code>pcre2_funcs</code> module for <a href="https://www.pcre.org">PCRE2</a> regular expressions.</p>
<h1 id="functions">Functions</h1>
<h2 id="matching">Matching</h2>
<h3 id="regexp">REGEXP()</h3>
<ul>
<li>REGEXP(re, string)</li>
<li>EXT_REGEXP(re, string)</li>
<li>string REGEXP re</li>
</ul>
<p>Returns 1 if <code>string</code> matches against the Extended Regular Expression <code>re</code>.</p>
<h3 id="basic_regexp">BASIC_REGEXP()</h3>
<ul>
<li>BASIC_REGEXP(re, string)</li>
</ul>
<p>Returns 1 if <code>string</code> matches against the Basic Regular Expression <code>re</code>.</p>
<h1 id="to-do">To-Do</h1>
<ul>
<li>Add the rest of the MySQL RE functions.</li>
</ul>
</body>
</html>



================================================
FILE: docs/posix_re_funcs.md
================================================
% POSIX regular expression functions

Introduction
============

This Sqlite3 extension module provides regular expression functions
using POSIX Extended and Basic syntax.

See the `string_funcs` module for versions using [ICU] regular
expressions and `pcre2_funcs` module for [PCRE2] regular expressions.

[PCRE2]: https://www.pcre.org
[ICU]: http://site.icu-project.org

Functions
=========

Matching
--------

### REGEXP()

* REGEXP(re, string)
* EXT_REGEXP(re, string)
* string REGEXP re

Returns 1 if `string` matches against the Extended Regular Expression `re`. 

### BASIC_REGEXP()

* BASIC_REGEXP(re, string)

Returns 1 if `string` matches against the Basic Regular Expression `re`.

To-Do
=====

* Add the rest of the MySQL RE functions.



================================================
FILE: docs/string_funcs.html
================================================
<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml" lang="" xml:lang="">
<head>
  <meta charset="utf-8" />
  <meta name="generator" content="pandoc" />
  <meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes" />
  <title>Extra String Functions</title>
  <style type="text/css">
      code{white-space: pre-wrap;}
      span.smallcaps{font-variant: small-caps;}
      span.underline{text-decoration: underline;}
      div.column{display: inline-block; vertical-align: top; width: 50%;}
  </style>
  <!--[if lt IE 9]>
    <script src="//cdnjs.cloudflare.com/ajax/libs/html5shiv/3.7.3/html5shiv-printshiv.min.js"></script>
  <![endif]-->
</head>
<body>
<header>
<h1 class="title">Extra String Functions</h1>
</header>
<nav id="TOC">
<ul>
<li><a href="#introduction">Introduction</a></li>
<li><a href="#scalar-functions">Scalar Functions</a><ul>
<li><a href="#informative">Informative</a></li>
<li><a href="#case-mapping">Case Mapping</a></li>
<li><a href="#text-extraction">Text Extraction</a></li>
<li><a href="#normalization">Normalization</a></li>
<li><a href="#other-conversions">Other conversions</a></li>
<li><a href="#unicode-text-compression">Unicode Text Compression</a></li>
<li><a href="#regular-expressions">Regular Expressions</a></li>
<li><a href="#other-functions">Other functions</a></li>
</ul></li>
<li><a href="#collations">Collations</a><ul>
<li><a href="#functions">Functions</a></li>
<li><a href="#predefined-collation-types">Predefined collation types</a></li>
<li><a href="#examples">Examples</a></li>
</ul></li>
</ul>
</nav>
<h1 id="introduction">Introduction</h1>
<p>This Sqlite3 extension module adds extra functionality for dealing with text in SQL queries, with an emphasis on Unicode. It depends on <a href="http://site.icu-project.org/">ICU</a> for most of its features. Many functions are inspired by MySQL and Postgresql string functions.</p>
<p>Since the standard <a href="https://www.sqlite.org/src/dir?ci=cdb68d2c64e453fd&amp;name=ext/icu">ICU extension</a> itself doesn’t seem to be compiled into or otherwise provided by many OS’s sqlite3 packages, it is included as part of this one. See that documentation for details about what it provides.</p>
<p>If you do a lot of things in your queries with Unicode text, or even just use Unicode-aware collations on index columns, consider setting the encoding of your databases to UTF-16 when creating them. Most <a href="http://site.icu-project.org/">ICU</a> functions work on UTF-16 strings, so this reduces the amount of converting to and from UTF-8.</p>
<h1 id="scalar-functions">Scalar Functions</h1>
<h2 id="informative">Informative</h2>
<h3 id="icu_version">ICU_VERSION()</h3>
<ul>
<li>ICU_VERSION()</li>
</ul>
<p>Returns the version of the ICU library being used.</p>
<h3 id="unicode_version">UNICODE_VERSION()</h3>
<ul>
<li>UNICODE_VERSION()</li>
</ul>
<p>Returns the version of Unicode understood by ICU.</p>
<h3 id="char_name">CHAR_NAME()</h3>
<ul>
<li>CHAR_NAME(c)</li>
</ul>
<p>Returns the Unicode name of the first codepoint in <code>c</code>, which can be an integer or string.</p>
<h3 id="script_name">SCRIPT_NAME()</h3>
<ul>
<li>SCRIPT_NAME(c)</li>
</ul>
<p>Returns the Unicode script of the first codepoint in <code>c</code>, which can be an integer or string.</p>
<h3 id="gclength">GCLENGTH()</h3>
<ul>
<li>GCLENGTH(string)</li>
<li>GCLENGTH(string, locale)</li>
</ul>
<p>Returns the number of <a href="http://unicode.org/reports/tr29/#Grapheme_Cluster_Boundaries">extended grapheme clusters</a> in <code>string</code>. This will be less than or equal to <code>LENGTH(string)</code>, which returns the number of <em>code points</em>.</p>
<h2 id="case-mapping">Case Mapping</h2>
<h3 id="upper">UPPER()</h3>
<ul>
<li>UPPER(string)</li>
<li>UPPER(string, locale)</li>
</ul>
<p>Returns <code>string</code> converted to uppercase, with an optional <code>locale</code> for specific rules.</p>
<h3 id="lower">LOWER()</h3>
<ul>
<li>LOWER(string)</li>
<li>LOWER(string, locale)</li>
</ul>
<p>Returns <code>string</code> converted to lowercase, with an optional <code>locale</code> for specific rules.</p>
<h3 id="title">TITLE()</h3>
<ul>
<li>TITLE(string)</li>
<li>TITLE(string, locale)</li>
<li>INITCAP(string)</li>
</ul>
<p>Returns <code>string</code> converted to lowercase, and the first letter of each word titlecased. The optional <code>locale</code> argument uses specific casing rules, like with <code>UPPER()</code> and <code>LOWER()</code>.</p>
<p>If <code>string</code> is <code>NULL</code>, returns <code>NULL</code>.</p>
<h3 id="casefold">CASEFOLD()</h3>
<ul>
<li>CASEFOLD(string)</li>
</ul>
<p>Returns a case-folded version of <code>string</code>.</p>
<p>If <code>string</code> is <code>NULL</code>, returns <code>NULL</code>.</p>
<h2 id="text-extraction">Text Extraction</h2>
<p>Sqlite3 provides one function, <code>SUBSTR()</code>, for extracting text from a string. It has the major drawback that it treats one Unicode code point as one character. As soon as you start getting outside of the Latin characters (And even in them if dealing with text in NFD format), that’s not true. It’s very easy to cut off a base character’s following combining characters, for example, with undesirable results.</p>
<p>The entire question of “What is a character?” gets very complicated fast when it comes to Unicode. The following functions consider characters to be <a href="http://unicode.org/reports/tr29/#Grapheme_Cluster_Boundaries">extended grapheme clusters</a>, which means they <em>usually</em> do what people expect.</p>
<h3 id="scalar-functions-1">Scalar Functions</h3>
<h4 id="gcleft">GCLEFT()</h4>
<ul>
<li>GCLEFT(string, len)</li>
<li>GCLEFT(string, len, locale)</li>
</ul>
<p>Returns the first <code>len</code> <a href="http://unicode.org/reports/tr29/#Grapheme_Cluster_Boundaries">extended grapheme clusters</a> from <code>string</code>.</p>
<p>If <code>len</code> is negative, returns all but the last <code>abs(len)</code> clusters.</p>
<h4 id="gcright">GCRIGHT()</h4>
<ul>
<li>GCRIGHT(string, len)</li>
<li>GCRIGHT(string, len, locale)</li>
</ul>
<p>Returns the last <code>len</code> <a href="http://unicode.org/reports/tr29/#Grapheme_Cluster_Boundaries">extended grapheme clusters</a> from <code>string</code>.</p>
<p>If <code>len</code> is negative, returns all but the first <code>abs(len)</code> clusters.</p>
<h4 id="gcsubstr">GCSUBSTR()</h4>
<ul>
<li>GCSUBSTR(string, start, len)</li>
<li>GCSUBSTR(string, start)</li>
<li>GCSUBSTR(string, start, len, locale)</li>
</ul>
<p>The <code>GCSUBSTR(string, start, len)</code> function returns a substring of input <code>string</code> that begins with the <code>start</code>-th extended grapheme cluster and which is <code>len</code> clusters long. If <code>len</code> is omitted or -1 then <code>GCSUBSTR(string, start)</code> returns all clusters through the end of the string beginning with the <code>start</code>-th. The left-most cluster of <code>string</code> is number 1.</p>
<h3 id="table-valued-functions">Table Valued Functions</h3>
<p>These functions use Unicode breaking algorithms from UAX#29 to split a string into its component tokens, one row per token. Each row has three columns: <code>value</code>, a string holding the token, <code>start</code>, the offset of the starting code point in the original string (The first codepoint is index 1), and <code>len</code>, the length of the token in code points. <code>substr(original, start, len)</code> will thus equal <code>value</code>.</p>
<h4 id="graphemes">GRAPHEMES()</h4>
<ul>
<li>GRAPHEMES(string)</li>
<li>GRAPHEMES(string, locale)</li>
</ul>
<p>Splits its argument up into individual extended grapheme clusters, optionally using a specific locale’s rules.</p>
<h4 id="words">WORDS()</h4>
<ul>
<li>WORDS(string)</li>
<li>WORDS(string, locale)</li>
</ul>
<p>Splits its argument into words, and the gaps betweens words, optionally using a specific locale’s rules.</p>
<p>To get only words, filter for only odd <code>rowid</code>s (Assuming the string starts with a word and not whitespace).</p>
<h4 id="sentences">SENTENCES()</h4>
<ul>
<li>SENTENCES(string)</li>
<li>SENTENCES(string, locale)</li>
</ul>
<p>Splits its argument into sentences.</p>
<h4 id="lines">LINES()</h4>
<ul>
<li>LINES(string)</li>
<li>LINES(string, locale)</li>
</ul>
<p>Splits its argument up into good spots for line breaks per UAX#14.</p>
<h2 id="normalization">Normalization</h2>
<p>Functions for normalizing Unicode text, and normalized concatentation (Since naive joining of two normalized Unicode strings can produce a non-normalized string. Yay Unicode!).</p>
<p>To-Do: Aggregate versions?</p>
<h3 id="normalize">NORMALIZE()</h3>
<ul>
<li>NORMALIZE(string, form)</li>
</ul>
<p>Returns <code>string</code> normalized according to <code>form</code>, which can be one of <code>'NFC'</code>, <code>'NFD'</code>, <code>'NFKC'</code>, <code>'NFKD'</code>, or <code>'NFKCCaseFold'</code>.</p>
<p>If <code>string</code> is <code>NULL</code>, returns <code>NULL</code>.</p>
<h3 id="nfc">NFC()</h3>
<ul>
<li>NFC(string, …)</li>
<li>NFC_WS(sep, string, …)</li>
</ul>
<p>Concatenates its non-NULL arguments together and returns the result in NFC. With one argument is equivalent to <code>NORMALIZE(string, 'NFC')</code>.</p>
<p>The <code>_WS</code> version intersperses <code>sep</code> between strings.</p>
<h3 id="nfd">NFD()</h3>
<ul>
<li>NFD(string, …)</li>
<li>NFD_WS(sep, string, …)</li>
</ul>
<p>Concatenates its non-NULL arguments together and returns the result in NFD. With one argument is equivalent to <code>NORMALIZE(string, 'NFD')</code>.</p>
<p>The <code>_WS</code> version intersperses <code>sep</code> between strings.</p>
<h3 id="nfkc">NFKC()</h3>
<ul>
<li>NFKC(string, …)</li>
<li>NFKC_WS(sep, string, …)</li>
</ul>
<p>Concatenates its non-NULL arguments together and returns the result in NFKC. With one argument is equivalent to <code>NORMALIZE(string, 'NFKC')</code>.</p>
<p>The <code>_WS</code> version intersperses <code>sep</code> between strings.</p>
<h3 id="nfkd">NFKD()</h3>
<ul>
<li>NFKD(string, …)</li>
<li>NFKD_WS(sep, string, …)</li>
</ul>
<p>Concatenates its non-NULL arguments together and returns the result in NFKD. With one argument is equivalent to <code>NORMALIZE(string, 'NFKD')</code>.</p>
<p>The <code>_WS</code> version intersperses <code>sep</code> between strings.</p>
<h2 id="other-conversions">Other conversions</h2>
<h3 id="to_ascii">TO_ASCII()</h3>
<ul>
<li>TO_ASCII(string)</li>
</ul>
<p>An enhanced version of <code>SPELLFIX1_TRANSLIT()</code> from the <em>spellfix1</em> extension. It converts Unicode text to ASCII, trying to gracefully downgrade many Latin accented characters and ligatures, transliterate Greek and Cyrillic characters, smart quotes, smart dashes, etc. It knows about more conversions than its inspiration, can handle characters outside the BMP, and deals with combining characters in a more intelligent way.</p>
<h3 id="convert_to">CONVERT_TO()</h3>
<ul>
<li>CONVERT_TO(string, charset)</li>
<li>CONVERT_TO(string, charset, substitution)</li>
</ul>
<p>Convert a Unicode string to the given character encoding, and return the result as a blob. The optional <code>substitution</code> string is used to replace characters that can’t be represented in the target encoding.</p>
<h3 id="convert_from">CONVERT_FROM()</h3>
<ul>
<li>CONVERT_FROM(blob, charset)</li>
</ul>
<p>Treats <code>blob</code> as being encoded in the given character encoding, and returns it converted to a Unicode string.</p>
<h3 id="strptime">STRPTIME()</h3>
<ul>
<li>STRPTIME(time-format, time-string)</li>
</ul>
<p>Wrapper for the C <code>strptime()</code> function. Returns a unixepoch time, or <code>null</code> on errors.</p>
<p>Example:</p>
<pre><code>SELECT date(strptime(&#39;%m/%d/%Y&#39;, &#39;10/08/2019&#39;), &#39;unixepoch&#39;);
-&gt; 2019-10-08</code></pre>
<h2 id="unicode-text-compression">Unicode Text Compression</h2>
<p>There are a few Unicode-specific text compression algorithms. They don’t have as good a compression ratio as more general purpose ones, but they have low overhead for compressing short strings. Could come in handy if you have a table with many short to medium length strings and are trying to save some space.</p>
<h3 id="scsu_compress">SCSU_COMPRESS()</h3>
<ul>
<li>SCSU_COMPRESS(string)</li>
</ul>
<p>Returns a blob representing <code>string</code> compressed with <a href="https://en.wikipedia.org/wiki/Standard_Compression_Scheme_for_Unicode">SCSU</a>.</p>
<h3 id="scsu_decompress">SCSU_DECOMPRESS()</h3>
<ul>
<li>SCSU_DECOMPRESS(blob)</li>
</ul>
<p>Decompresses <code>blob</code>, which should be <a href="https://en.wikipedia.org/wiki/Standard_Compression_Scheme_for_Unicode">SCSU</a> compressed Unicode text.</p>
<h3 id="bocu_compress">BOCU_COMPRESS()</h3>
<ul>
<li>BOCU_COMPRESS(string)</li>
</ul>
<p>Returns a blob representing <code>string</code> compressed with <a href="https://en.wikipedia.org/wiki/Binary_Ordered_Compression_for_Unicode">BOCU-1</a>.</p>
<h3 id="bocu_decompress">BOCU_DECOMPRESS()</h3>
<ul>
<li>BOCU_DECOMPRESS(blob)</li>
</ul>
<p>Decompresses <code>blob</code>, which should be <a href="https://en.wikipedia.org/wiki/Binary_Ordered_Compression_for_Unicode">BOCU-1</a> compressed Unicode text.</p>
<h2 id="regular-expressions">Regular Expressions</h2>
<p>MySQL-compatible regular expression functions. All also work when the name is prefixed by <strong>ICU_</strong>, to support coexisting with future extensions that use different engines - PCRE, RE2, etc. being loaded at the same time.</p>
<p>The <code>match_type</code> string argument supports some extra options over MySQL:</p>
<ul>
<li><em>w</em> means to use Unicode word breaks instead of traditional ones.</li>
<li><em>x</em> means that the regexp can have comments and whitespace.</li>
<li><em>l</em> means to treat the regexp as a literal string to search for and not a regular expression.</li>
</ul>
<h3 id="regexp">REGEXP()</h3>
<ul>
<li>REGEXP(re, string)</li>
<li>REGEXP(re, string, match_type)</li>
<li>string REGEXP re</li>
</ul>
<p>Returns 1 if <code>string</code> matches <code>re</code>. The entire string must match. The three argument version is an extension over the normal ICU extension implementation.</p>
<h3 id="regexp_instr">REGEXP_INSTR()</h3>
<p>See <a href="https://dev.mysql.com/doc/refman/8.0/en/regexp.html#function_regexp-instr">MySQL REGEXP_INSTR()</a> documentation.</p>
<p>If the <code>match_type</code> option has a digit in the range 0-9 in it, the position of that capturing group is returned instead of the complete match. 0 is the full match.</p>
<h3 id="regexp_like">REGEXP_LIKE()</h3>
<p>See <a href="https://dev.mysql.com/doc/refman/8.0/en/regexp.html#function_regexp-like">MySQL REGEXP_LIKE()</a> documentation.</p>
<h3 id="regexp_replace">REGEXP_REPLACE()</h3>
<p>See <a href="https://dev.mysql.com/doc/refman/8.0/en/regexp.html#function_regexp-replace">MySQL REGEXP_REPLACE()</a> documentation.</p>
<p>This implementation currently only supports a <code>pos</code> argument of 1 and <code>occurence</code> of 0 or 1. It also replaces tokens like <code>$N</code> in the replacement string with the N-th capture group.</p>
<h3 id="regexp_substr">REGEXP_SUBSTR()</h3>
<p>See <a href="https://dev.mysql.com/doc/refman/8.0/en/regexp.html#function_regexp-substr">MySQL REGEXP_SUBSTR()</a> documentation.</p>
<p>If the <code>match_type</code> option has a digit in the range 0-9 in it, that capturing group is returned instead of the complete match. 0 is the full match.</p>
<h2 id="other-functions">Other functions</h2>
<h3 id="concat">CONCAT()</h3>
<ul>
<li>CONCAT(string, …)</li>
<li>MYSQL_CONCAT(string, …)</li>
<li>CONCAT_WS(sep, string, …)</li>
</ul>
<p>Returns a string concatenating its arguments together. If <code>MYSQL_CONCAT()</code> gets a <code>NULL</code> argument, it returns <code>NULL</code>. The other versions just skip those arguments. The <code>_WS</code> version puts <code>sep</code> between strings.</p>
<h3 id="repeat">REPEAT()</h3>
<ul>
<li>REPEAT(string, count)</li>
<li>REPEAT(string, count, form)</li>
</ul>
<p>Returns a new string created by repeating <code>string</code> <code>count</code> times. If a third argument is given, it’s the normalization form to use for the result.</p>
<h3 id="confusable">CONFUSABLE()</h3>
<ul>
<li>CONFUSABLE(string1, string2)</li>
</ul>
<p>Returns 1 if its two arguments are the same or can easily be visually mistaken to be the same, 0 if they’re distinct.</p>
<p>(If only one of the arguments is a string literal or bound to a placeholder, it should be the first one.)</p>
<h1 id="collations">Collations</h1>
<p>For when <code>BINARY</code> and <code>NOCASE</code> aren’t good enough.</p>
<h2 id="functions">Functions</h2>
<h3 id="icu_load_collationlocale-name">ICU_LOAD_COLLATION(locale, name)</h3>
<p>See the <a href="https://www.sqlite.org/src/dir?ci=cdb68d2c64e453fd&amp;name=ext/icu">ICU extension</a> documentation.</p>
<h2 id="predefined-collation-types">Predefined collation types</h2>
<h3 id="codepoint">CODEPOINT</h3>
<p>Compares code points instead of code units like <code>BINARY</code> does. Makes a difference when comparing UTF-16 text with surrogate pairs.</p>
<h3 id="unocase">UNOCASE</h3>
<p>Unicode-aware case-insensitive ordering. Compares case-folded code points without any locale-specific rules. If doing lots of comparisions, it’s better to use precomputed casefolded strings.</p>
<h3 id="equiv">EQUIV</h3>
<p>Unicode equivalence. The same string normalized in two different forms is equivalent. If comparing a lot of strings, it’s best to canonize them with the same normalization form.</p>
<h3 id="enocase">ENOCASE</h3>
<p>Case-insensitive Unicode equivalence. If comparing a lot of strings, it’s better to use precomputed case folded and normalized ones.</p>
<h2 id="examples">Examples</h2>
<pre><code>char(0x0122) = char(0x0123) COLLATE BINARY       =&gt; 0
char(0x0122) = char(0x0123) COLLATE NOCASE       =&gt; 0
char(0x0122) = char(0x0123) COLLATE UNOCASE      =&gt; 1
char(0x0122) = nfd(char(0x0122)) COLATE BINARY   =&gt; 0
char(0x0122) = nfd(char(0x0122)) COLLATE EQUIV   =&gt; 1
char(0x0122) = nfd(char(0x0123)) COLLATE EQUIV   =&gt; 0
char(0x0122) = nfd(char(0x0123)) COLLATE ENOCASE =&gt; 1</code></pre>
</body>
</html>



================================================
FILE: docs/string_funcs.md
================================================
% Extra String Functions

Introduction
============

This Sqlite3 extension module adds extra functionality for dealing
with text in SQL queries, with an emphasis on Unicode. It depends on
[ICU] for most of its features. Many functions are inspired by MySQL
and Postgresql string functions.

Since the standard [ICU extension] itself doesn't seem to be compiled
into or otherwise provided by many OS's sqlite3 packages, it is
included as part of this one. See that documentation for details about
what it provides.

If you do a lot of things in your queries with Unicode text, or even
just use Unicode-aware collations on index columns, consider setting
the encoding of your databases to UTF-16 when creating them. Most
[ICU] functions work on UTF-16 strings, so this reduces the amount of
converting to and from UTF-8.

[ICU]: http://site.icu-project.org/
[ICU extension]: https://www.sqlite.org/src/dir?ci=cdb68d2c64e453fd&name=ext/icu

Scalar Functions
================

Informative
-----------

### ICU_VERSION()

* ICU_VERSION()

Returns the version of the ICU library being used.

### UNICODE_VERSION()

* UNICODE_VERSION()

Returns the version of Unicode understood by ICU.

### CHAR_NAME()

* CHAR_NAME(c)

Returns the Unicode name of the first codepoint in `c`, which can be
an integer or string.

### SCRIPT_NAME()

* SCRIPT_NAME(c)

Returns the Unicode script of the first codepoint in `c`, which can be
an integer or string.

### GCLENGTH()

* GCLENGTH(string)
* GCLENGTH(string, locale)

Returns the number of [extended grapheme clusters] in `string`. This
will be less than or equal to `LENGTH(string)`, which returns the
number of *code points*.

[extended grapheme clusters]: http://unicode.org/reports/tr29/#Grapheme_Cluster_Boundaries

Case Mapping
------------

### UPPER()

* UPPER(string)
* UPPER(string, locale)

Returns `string` converted to uppercase, with an optional `locale` for
specific rules.

### LOWER()

* LOWER(string)
* LOWER(string, locale)

Returns `string` converted to lowercase, with an optional `locale` for
specific rules.

### TITLE()

* TITLE(string)
* TITLE(string, locale)
* INITCAP(string)

Returns `string` converted to lowercase, and the first letter of each
word titlecased. The optional `locale` argument uses specific casing
rules, like with `UPPER()` and `LOWER()`.

If `string` is `NULL`, returns `NULL`.

### CASEFOLD()

* CASEFOLD(string)

Returns a case-folded version of `string`.

If `string` is `NULL`, returns `NULL`.

Text Extraction
---------------

Sqlite3 provides one function, `SUBSTR()`, for extracting text from a
string. It has the major drawback that it treats one Unicode code
point as one character. As soon as you start getting outside of the
Latin characters (And even in them if dealing with text in NFD
format), that's not true. It's very easy to cut off a base character's
following combining characters, for example, with undesirable results.

The entire question of "What is a character?" gets very complicated
fast when it comes to Unicode. The following functions consider
characters to be [extended grapheme clusters], which means they
*usually* do what people expect.

### Scalar Functions

#### GCLEFT()

* GCLEFT(string, len)
* GCLEFT(string, len, locale)

Returns the first `len` [extended grapheme clusters] from `string`.

If `len` is negative, returns all but the last `abs(len)` clusters.

#### GCRIGHT()

* GCRIGHT(string, len)
* GCRIGHT(string, len, locale)

Returns the last `len` [extended grapheme clusters] from `string`.

If `len` is negative, returns all but the first `abs(len)` clusters.

#### GCSUBSTR()

* GCSUBSTR(string, start, len)
* GCSUBSTR(string, start)
* GCSUBSTR(string, start, len, locale)

The `GCSUBSTR(string, start, len)` function returns a substring of
input `string` that begins with the `start`-th extended grapheme
cluster and which is `len` clusters long. If `len` is omitted or -1 then
`GCSUBSTR(string, start)` returns all clusters through the end of the
string beginning with the `start`-th. The left-most cluster of
`string` is number 1.

### Table Valued Functions

These functions use Unicode breaking algorithms from UAX#29 to split a
string into its component tokens, one row per token. Each row has
three columns: `value`, a string holding the token, `start`, the
offset of the starting code point in the original string (The first
codepoint is index 1), and `len`, the length of the token in code
points. `substr(original, start, len)` will thus equal `value`.

#### GRAPHEMES()

* GRAPHEMES(string)
* GRAPHEMES(string, locale)

Splits its argument up into individual extended grapheme clusters,
optionally using a specific locale's rules.

#### WORDS()

* WORDS(string)
* WORDS(string, locale)

Splits its argument into words, and the gaps betweens words,
optionally using a specific locale's rules.

To get only words, filter for only odd `rowid`s (Assuming the string
starts with a word and not whitespace).

#### SENTENCES()

* SENTENCES(string)
* SENTENCES(string, locale)

Splits its argument into sentences.

#### LINES()

* LINES(string)
* LINES(string, locale)

Splits its argument up into good spots for line breaks per UAX#14.


Normalization
-------------

Functions for normalizing Unicode text, and normalized concatentation
(Since naive joining of two normalized Unicode strings can produce a
non-normalized string. Yay Unicode!).

To-Do: Aggregate versions?

### NORMALIZE()

* NORMALIZE(string, form)

Returns `string` normalized according to `form`, which can be one of
`'NFC'`, `'NFD'`, `'NFKC'`, `'NFKD'`, or `'NFKCCaseFold'`.

If `string` is `NULL`, returns `NULL`.

### NFC()

* NFC(string, ...)
* NFC_WS(sep, string, ...)

Concatenates its non-NULL arguments together and returns the result in
NFC. With one argument is equivalent to `NORMALIZE(string, 'NFC')`.

The `_WS` version intersperses `sep` between strings.

### NFD()

* NFD(string, ...)
* NFD_WS(sep, string, ...)

Concatenates its non-NULL arguments together and returns the result in
NFD. With one argument is equivalent to `NORMALIZE(string, 'NFD')`.

The `_WS` version intersperses `sep` between strings.

### NFKC()

* NFKC(string, ...)
* NFKC_WS(sep, string, ...)

Concatenates its non-NULL arguments together and returns the result in
NFKC. With one argument is equivalent to `NORMALIZE(string, 'NFKC')`.

The `_WS` version intersperses `sep` between strings.

### NFKD()

* NFKD(string, ...)
* NFKD_WS(sep, string, ...)

Concatenates its non-NULL arguments together and returns the result in
NFKD. With one argument is equivalent to `NORMALIZE(string, 'NFKD')`.

The `_WS` version intersperses `sep` between strings.

Other conversions
-----------------

### TO_ASCII()

* TO_ASCII(string)

An enhanced version of `SPELLFIX1_TRANSLIT()` from the *spellfix1*
extension. It converts Unicode text to ASCII, trying to gracefully
downgrade many Latin accented characters and ligatures, transliterate
Greek and Cyrillic characters, smart quotes, smart dashes, etc. It
knows about more conversions than its inspiration, can handle
characters outside the BMP, and deals with combining characters in a
more intelligent way.

### CONVERT_TO()

* CONVERT_TO(string, charset)
* CONVERT_TO(string, charset, substitution)

Convert a Unicode string to the given character encoding, and return
the result as a blob. The optional `substitution` string is used to
replace characters that can't be represented in the target encoding.

### CONVERT_FROM()

* CONVERT_FROM(blob, charset)

Treats `blob` as being encoded in the given character encoding, and
returns it converted to a Unicode string.

### STRPTIME()

* STRPTIME(time-format, time-string)

Wrapper for the C `strptime()` function. Returns a unixepoch time, or
`null` on errors.

Example:

    SELECT date(strptime('%m/%d/%Y', '10/08/2019'), 'unixepoch');
    -> 2019-10-08
    

Unicode Text Compression
------------------------

There are a few Unicode-specific text compression algorithms. They
don't have as good a compression ratio as more general purpose ones,
but they have low overhead for compressing short strings. Could come
in handy if you have a table with many short to medium length strings
and are trying to save some space.

### SCSU_COMPRESS()

* SCSU_COMPRESS(string)

Returns a blob representing `string` compressed with [SCSU].

### SCSU_DECOMPRESS()

* SCSU_DECOMPRESS(blob)

Decompresses `blob`, which should be [SCSU] compressed Unicode text.

### BOCU_COMPRESS()

* BOCU_COMPRESS(string)

Returns a blob representing `string` compressed with [BOCU-1].

### BOCU_DECOMPRESS()

* BOCU_DECOMPRESS(blob)

Decompresses `blob`, which should be [BOCU-1] compressed Unicode text.

[SCSU]: https://en.wikipedia.org/wiki/Standard_Compression_Scheme_for_Unicode
[BOCU-1]: https://en.wikipedia.org/wiki/Binary_Ordered_Compression_for_Unicode

Regular Expressions
-------------------

MySQL-compatible regular expression functions. All also work when the
name is prefixed by **ICU\_**, to support coexisting with future
extensions that use different engines - PCRE, RE2, etc. being loaded
at the same time.

The `match_type` string argument supports some extra options over MySQL:

* *w* means to use Unicode word breaks instead of traditional ones.
* *x* means that the regexp can have comments and whitespace.
* *l* means to treat the regexp as a literal string to search for and
  not a regular expression.

### REGEXP()

* REGEXP(re, string)
* REGEXP(re, string, match_type)
* string REGEXP re

Returns 1 if `string` matches `re`. The entire string must match. The
three argument version is an extension over the normal ICU extension
implementation.

### REGEXP_INSTR()

See [MySQL REGEXP_INSTR()] documentation.

If the `match_type` option has a digit in the range 0-9 in it, the
position of that capturing group is returned instead of the complete
match. 0 is the full match.

[MySQL REGEXP_INSTR()]: https://dev.mysql.com/doc/refman/8.0/en/regexp.html#function_regexp-instr

### REGEXP_LIKE()

See [MySQL REGEXP_LIKE()] documentation.

[MySQL REGEXP_LIKE()]: https://dev.mysql.com/doc/refman/8.0/en/regexp.html#function_regexp-like

### REGEXP_REPLACE()

See [MySQL REGEXP_REPLACE()] documentation.

This implementation currently only supports a `pos` argument of 1 and
`occurence` of 0 or 1. It also replaces tokens like `$N` in the
replacement string with the N-th capture group.

[MySQL REGEXP_REPLACE()]: https://dev.mysql.com/doc/refman/8.0/en/regexp.html#function_regexp-replace

### REGEXP_SUBSTR()

See [MySQL REGEXP_SUBSTR()] documentation.

If the `match_type` option has a digit in the range 0-9 in it, that
capturing group is returned instead of the complete match. 0 is the
full match.

[MySQL REGEXP_SUBSTR()]: https://dev.mysql.com/doc/refman/8.0/en/regexp.html#function_regexp-substr

Other functions
---------------

### CONCAT()

* CONCAT(string, ...)
* MYSQL_CONCAT(string, ...)
* CONCAT_WS(sep, string, ...)

Returns a string concatenating its arguments together. If
`MYSQL_CONCAT()` gets a `NULL` argument, it returns `NULL`. The other
versions just skip those arguments. The `_WS` version puts `sep`
between strings.

### REPEAT()

* REPEAT(string, count)
* REPEAT(string, count, form)

Returns a new string created by repeating `string` `count` times. If a
third argument is given, it's the normalization form to use for the
result.

### CONFUSABLE()

* CONFUSABLE(string1, string2)

Returns 1 if its two arguments are the same or can easily be visually
mistaken to be the same, 0 if they're distinct.

(If only one of the arguments is a string literal or bound to a
placeholder, it should be the first one.)

Collations
==========

For when `BINARY` and `NOCASE` aren't good enough.

Functions
---------

### ICU_LOAD_COLLATION(locale, name)

See the [ICU extension] documentation.

Predefined collation types
--------------------------

### CODEPOINT

Compares code points instead of code units like `BINARY` does. Makes a
difference when comparing UTF-16 text with surrogate pairs.

### UNOCASE

Unicode-aware case-insensitive ordering. Compares case-folded code
points without any locale-specific rules. If doing lots of
comparisions, it's better to use precomputed casefolded strings.

### EQUIV

Unicode equivalence. The same string normalized in two
different forms is equivalent. If comparing a lot of strings, it's
best to canonize them with the same normalization form.

### ENOCASE

Case-insensitive Unicode equivalence. If comparing a lot of strings,
it's better to use precomputed case folded and normalized ones.

Examples
--------

    char(0x0122) = char(0x0123) COLLATE BINARY       => 0
    char(0x0122) = char(0x0123) COLLATE NOCASE       => 0
    char(0x0122) = char(0x0123) COLLATE UNOCASE      => 1
    char(0x0122) = nfd(char(0x0122)) COLATE BINARY   => 0
    char(0x0122) = nfd(char(0x0122)) COLLATE EQUIV   => 1
    char(0x0122) = nfd(char(0x0123)) COLLATE EQUIV   => 0
    char(0x0122) = nfd(char(0x0123)) COLLATE ENOCASE => 1




================================================
FILE: src/blob_funcs.c
================================================
/*
Copyright 2018-2019 Shawn Wagner

Permission is hereby granted, free of charge, to any person obtaining
a copy of this software and associated documentation files (the
"Software"), to deal in the Software without restriction, including
without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject to
the following conditions:

The above copyright notice and this permission notice shall be
included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/

#include <assert.h>
#include <ctype.h>
#include <limits.h>
#include <openssl/bio.h>
#include <openssl/err.h>
#include <openssl/evp.h>
#include <openssl/hmac.h>
#include <openssl/md5.h>
#include <openssl/rand.h>
#include <openssl/sha.h>
#include <sqlite3ext.h>
#include <string.h>

#if defined(__GNUC__) && (defined(__x86_64) || defined(__i386))
#include <nmmintrin.h>
#endif

#include "config.h"

#ifdef ZLIB_FOUND
#include <arpa/inet.h>
#include <zlib.h>
#endif

SQLITE_EXTENSION_INIT1

static char *to_hex(const unsigned char *bytes, int len) {
  static const char hexdigits[] = "0123456789ABCDEF";
  char *output = sqlite3_malloc(len * 2 + 1);
  if (!output) {
    return NULL;
  }
  int i = 0;
  for (int n = 0; n < len; n += 1) {
    output[i++] = hexdigits[bytes[n] >> 4];
    output[i++] = hexdigits[bytes[n] & 0xF];
  }
  output[i] = '\0';
  return output;
}

static void bf_md5(sqlite3_context *ctx, int nargs __attribute__((unused)),
                   sqlite3_value **args) {
  if (sqlite3_value_type(args[0]) == SQLITE_NULL) {
    return;
  }

  const unsigned char *data = sqlite3_value_blob(args[0]);
  if (!data) {
    return;
  }
  int datalen = sqlite3_value_bytes(args[0]);
  unsigned char md[MD5_DIGEST_LENGTH];
  if (!MD5(data, datalen, md)) {
    sqlite3_result_error(ctx, "MD5 failed", -1);
    return;
  }
  char *hex = to_hex(md, MD5_DIGEST_LENGTH);
  if (!hex) {
    sqlite3_result_error_nomem(ctx);
    return;
  }

  sqlite3_result_text(ctx, hex, MD5_DIGEST_LENGTH * 2, sqlite3_free);
}

static void bf_sha1(sqlite3_context *ctx, int nargs __attribute__((unused)),
                    sqlite3_value **args) {
  if (sqlite3_value_type(args[0]) == SQLITE_NULL) {
    return;
  }

  const unsigned char *data = sqlite3_value_blob(args[0]);
  if (!data) {
    return;
  }
  int datalen = sqlite3_value_bytes(args[0]);
  unsigned char md[SHA_DIGEST_LENGTH];
  if (!SHA1(data, datalen, md)) {
    sqlite3_result_error(ctx, "SHA1 failed", -1);
    return;
  }
  char *hex = to_hex(md, SHA_DIGEST_LENGTH);
  if (!hex) {
    sqlite3_result_error_nomem(ctx);
    return;
  }
  sqlite3_result_text(ctx, hex, SHA_DIGEST_LENGTH * 2, sqlite3_free);
}

static const EVP_MD *get_sha2_algo(int bits) {
  switch (bits) {
  case 224:
    return EVP_sha224();
  case 0:
  case 256:
    return EVP_sha256();
  case 384:
    return EVP_sha384();
  case 512:
    return EVP_sha512();
  default:
    return NULL;
  }
}

static void bf_sha2(sqlite3_context *ctx, int nargs, sqlite3_value **args) {

  if (sqlite3_value_type(args[0]) == SQLITE_NULL) {
    return;
  }
  if (nargs == 2 && sqlite3_value_type(args[1]) == SQLITE_NULL) {
    return;
  }
  const unsigned char *data = sqlite3_value_blob(args[0]);
  if (!data) {
    return;
  }
  int datalen = sqlite3_value_bytes(args[0]);

  int bits = 0;
  if (nargs == 2) {
    bits = sqlite3_value_int(args[1]);
  }

  const EVP_MD *algo = get_sha2_algo(bits);
  if (!algo) {
    return;
  }

  unsigned char md[EVP_MAX_MD_SIZE];
  unsigned int mdlen;
  EVP_MD_CTX *hashctx;

#ifdef HAVE_EVP_MD_CTX_NEW
  hashctx = EVP_MD_CTX_new();
#else
  hashctx = EVP_MD_CTX_create();
#endif
  if (!hashctx) {
    sqlite3_result_error_nomem(ctx);
    return;
  }
  if (!EVP_DigestInit_ex(hashctx, algo, NULL)) {
    sqlite3_result_error(ctx, "EVP_DigestInit_ex failed", -1);
#ifdef HAVE_EVP_MD_CTX_NEW
    EVP_MD_CTX_free(hashctx);
#else
    EVP_MD_CTX_destroy(hashctx);
#endif
    return;
  }
  if (!EVP_DigestUpdate(hashctx, data, datalen)) {
    sqlite3_result_error(ctx, "EVP_DigestUpdate failed", -1);
#ifdef HAVE_EVP_MD_CTX_NEW
    EVP_MD_CTX_free(hashctx);
#else
    EVP_MD_CTX_destroy(hashctx);
#endif
    return;
  }
  if (!EVP_DigestFinal_ex(hashctx, md, &mdlen)) {
    sqlite3_result_error(ctx, "EVP_DigestFinal_ex failed", -1);
#ifdef HAVE_EVP_MD_CTX_NEW
    EVP_MD_CTX_free(hashctx);
#else
    EVP_MD_CTX_destroy(hashctx);
#endif
    return;
  }
#ifdef HAVE_EVP_MD_CTX_NEW
  EVP_MD_CTX_free(hashctx);
#else
  EVP_MD_CTX_destroy(hashctx);
#endif

  char *hex = to_hex(md, mdlen);
  if (!hex) {
    sqlite3_result_error_nomem(ctx);
    return;
  }
  sqlite3_result_text(ctx, hex, mdlen * 2, sqlite3_free);
}

// Start of code taken from sqlite3 shathree.c

/*
** 2017-03-08
**
** The author disclaims copyright to this source code.  In place of
** a legal notice, here is a blessing:
**
**    May you do good and not evil.
**    May you find forgiveness for yourself and forgive others.
**    May you share freely, never taking more than you give.
**
******************************************************************************
**
** This SQLite extension implements a functions that compute SHA1 hashes.
** Two SQL functions are implemented:
**
**     sha3(X,SIZE)
**     sha3_query(Y,SIZE)
**
** The sha3(X) function computes the SHA3 hash of the input X, or NULL if
** X is NULL.
**
** The sha3_query(Y) function evalutes all queries in the SQL statements of Y
** and returns a hash of their results.
**
** The SIZE argument is optional.  If omitted, the SHA3-256 hash algorithm
** is used.  If SIZE is included it must be one of the integers 224, 256,
** 384, or 512, to determine SHA3 hash variant that is computed.
*/
#include <assert.h>
#include <stdarg.h>
#include <string.h>
typedef sqlite3_uint64 u64;

/******************************************************************************
** The Hash Engine
*/
/*
** Macros to determine whether the machine is big or little endian,
** and whether or not that determination is run-time or compile-time.
**
** For best performance, an attempt is made to guess at the byte-order
** using C-preprocessor macros.  If that is unsuccessful, or if
** -DSHA3_BYTEORDER=0 is set, then byte-order is determined
** at run-time.
*/
#ifndef SHA3_BYTEORDER
#if defined(i386) || defined(__i386__) || defined(_M_IX86) ||                  \
    defined(__x86_64) || defined(__x86_64__) || defined(_M_X64) ||             \
    defined(_M_AMD64) || defined(_M_ARM) || defined(__x86) || defined(__arm__)
#define SHA3_BYTEORDER 1234
#elif defined(sparc) || defined(__ppc__)
#define SHA3_BYTEORDER 4321
#else
#define SHA3_BYTEORDER 0
#endif
#endif

/*
** State structure for a SHA3 hash in progress
*/
typedef struct SHA3Context SHA3Context;
struct SHA3Context {
  union {
    u64 s[25];             /* Keccak state. 5x5 lines of 64 bits each */
    unsigned char x[1600]; /* ... or 1600 bytes */
  } u;
  unsigned nRate;   /* Bytes of input accepted per Keccak iteration */
  unsigned nLoaded; /* Input bytes loaded into u.x[] so far this cycle */
  unsigned ixMask;  /* Insert next input into u.x[nLoaded^ixMask]. */
};

/*
** A single step of the Keccak mixing function for a 1600-bit state
*/
static void KeccakF1600Step(SHA3Context *p) {
  int i;
  u64 b0, b1, b2, b3, b4;
  u64 c0, c1, c2, c3, c4;
  u64 d0, d1, d2, d3, d4;
  static const u64 RC[] = {
      0x0000000000000001ULL, 0x0000000000008082ULL, 0x800000000000808aULL,
      0x8000000080008000ULL, 0x000000000000808bULL, 0x0000000080000001ULL,
      0x8000000080008081ULL, 0x8000000000008009ULL, 0x000000000000008aULL,
      0x0000000000000088ULL, 0x0000000080008009ULL, 0x000000008000000aULL,
      0x000000008000808bULL, 0x800000000000008bULL, 0x8000000000008089ULL,
      0x8000000000008003ULL, 0x8000000000008002ULL, 0x8000000000000080ULL,
      0x000000000000800aULL, 0x800000008000000aULL, 0x8000000080008081ULL,
      0x8000000000008080ULL, 0x0000000080000001ULL, 0x8000000080008008ULL};
#define a00 (p->u.s[0])
#define a01 (p->u.s[1])
#define a02 (p->u.s[2])
#define a03 (p->u.s[3])
#define a04 (p->u.s[4])
#define a10 (p->u.s[5])
#define a11 (p->u.s[6])
#define a12 (p->u.s[7])
#define a13 (p->u.s[8])
#define a14 (p->u.s[9])
#define a20 (p->u.s[10])
#define a21 (p->u.s[11])
#define a22 (p->u.s[12])
#define a23 (p->u.s[13])
#define a24 (p->u.s[14])
#define a30 (p->u.s[15])
#define a31 (p->u.s[16])
#define a32 (p->u.s[17])
#define a33 (p->u.s[18])
#define a34 (p->u.s[19])
#define a40 (p->u.s[20])
#define a41 (p->u.s[21])
#define a42 (p->u.s[22])
#define a43 (p->u.s[23])
#define a44 (p->u.s[24])
#define ROL64(a, x) ((a << x) | (a >> (64 - x)))

  for (i = 0; i < 24; i += 4) {
    c0 = a00 ^ a10 ^ a20 ^ a30 ^ a40;
    c1 = a01 ^ a11 ^ a21 ^ a31 ^ a41;
    c2 = a02 ^ a12 ^ a22 ^ a32 ^ a42;
    c3 = a03 ^ a13 ^ a23 ^ a33 ^ a43;
    c4 = a04 ^ a14 ^ a24 ^ a34 ^ a44;
    d0 = c4 ^ ROL64(c1, 1);
    d1 = c0 ^ ROL64(c2, 1);
    d2 = c1 ^ ROL64(c3, 1);
    d3 = c2 ^ ROL64(c4, 1);
    d4 = c3 ^ ROL64(c0, 1);

    b0 = (a00 ^ d0);
    b1 = ROL64((a11 ^ d1), 44);
    b2 = ROL64((a22 ^ d2), 43);
    b3 = ROL64((a33 ^ d3), 21);
    b4 = ROL64((a44 ^ d4), 14);
    a00 = b0 ^ ((~b1) & b2);
    a00 ^= RC[i];
    a11 = b1 ^ ((~b2) & b3);
    a22 = b2 ^ ((~b3) & b4);
    a33 = b3 ^ ((~b4) & b0);
    a44 = b4 ^ ((~b0) & b1);

    b2 = ROL64((a20 ^ d0), 3);
    b3 = ROL64((a31 ^ d1), 45);
    b4 = ROL64((a42 ^ d2), 61);
    b0 = ROL64((a03 ^ d3), 28);
    b1 = ROL64((a14 ^ d4), 20);
    a20 = b0 ^ ((~b1) & b2);
    a31 = b1 ^ ((~b2) & b3);
    a42 = b2 ^ ((~b3) & b4);
    a03 = b3 ^ ((~b4) & b0);
    a14 = b4 ^ ((~b0) & b1);

    b4 = ROL64((a40 ^ d0), 18);
    b0 = ROL64((a01 ^ d1), 1);
    b1 = ROL64((a12 ^ d2), 6);
    b2 = ROL64((a23 ^ d3), 25);
    b3 = ROL64((a34 ^ d4), 8);
    a40 = b0 ^ ((~b1) & b2);
    a01 = b1 ^ ((~b2) & b3);
    a12 = b2 ^ ((~b3) & b4);
    a23 = b3 ^ ((~b4) & b0);
    a34 = b4 ^ ((~b0) & b1);

    b1 = ROL64((a10 ^ d0), 36);
    b2 = ROL64((a21 ^ d1), 10);
    b3 = ROL64((a32 ^ d2), 15);
    b4 = ROL64((a43 ^ d3), 56);
    b0 = ROL64((a04 ^ d4), 27);
    a10 = b0 ^ ((~b1) & b2);
    a21 = b1 ^ ((~b2) & b3);
    a32 = b2 ^ ((~b3) & b4);
    a43 = b3 ^ ((~b4) & b0);
    a04 = b4 ^ ((~b0) & b1);

    b3 = ROL64((a30 ^ d0), 41);
    b4 = ROL64((a41 ^ d1), 2);
    b0 = ROL64((a02 ^ d2), 62);
    b1 = ROL64((a13 ^ d3), 55);
    b2 = ROL64((a24 ^ d4), 39);
    a30 = b0 ^ ((~b1) & b2);
    a41 = b1 ^ ((~b2) & b3);
    a02 = b2 ^ ((~b3) & b4);
    a13 = b3 ^ ((~b4) & b0);
    a24 = b4 ^ ((~b0) & b1);

    c0 = a00 ^ a20 ^ a40 ^ a10 ^ a30;
    c1 = a11 ^ a31 ^ a01 ^ a21 ^ a41;
    c2 = a22 ^ a42 ^ a12 ^ a32 ^ a02;
    c3 = a33 ^ a03 ^ a23 ^ a43 ^ a13;
    c4 = a44 ^ a14 ^ a34 ^ a04 ^ a24;
    d0 = c4 ^ ROL64(c1, 1);
    d1 = c0 ^ ROL64(c2, 1);
    d2 = c1 ^ ROL64(c3, 1);
    d3 = c2 ^ ROL64(c4, 1);
    d4 = c3 ^ ROL64(c0, 1);

    b0 = (a00 ^ d0);
    b1 = ROL64((a31 ^ d1), 44);
    b2 = ROL64((a12 ^ d2), 43);
    b3 = ROL64((a43 ^ d3), 21);
    b4 = ROL64((a24 ^ d4), 14);
    a00 = b0 ^ ((~b1) & b2);
    a00 ^= RC[i + 1];
    a31 = b1 ^ ((~b2) & b3);
    a12 = b2 ^ ((~b3) & b4);
    a43 = b3 ^ ((~b4) & b0);
    a24 = b4 ^ ((~b0) & b1);

    b2 = ROL64((a40 ^ d0), 3);
    b3 = ROL64((a21 ^ d1), 45);
    b4 = ROL64((a02 ^ d2), 61);
    b0 = ROL64((a33 ^ d3), 28);
    b1 = ROL64((a14 ^ d4), 20);
    a40 = b0 ^ ((~b1) & b2);
    a21 = b1 ^ ((~b2) & b3);
    a02 = b2 ^ ((~b3) & b4);
    a33 = b3 ^ ((~b4) & b0);
    a14 = b4 ^ ((~b0) & b1);

    b4 = ROL64((a30 ^ d0), 18);
    b0 = ROL64((a11 ^ d1), 1);
    b1 = ROL64((a42 ^ d2), 6);
    b2 = ROL64((a23 ^ d3), 25);
    b3 = ROL64((a04 ^ d4), 8);
    a30 = b0 ^ ((~b1) & b2);
    a11 = b1 ^ ((~b2) & b3);
    a42 = b2 ^ ((~b3) & b4);
    a23 = b3 ^ ((~b4) & b0);
    a04 = b4 ^ ((~b0) & b1);

    b1 = ROL64((a20 ^ d0), 36);
    b2 = ROL64((a01 ^ d1), 10);
    b3 = ROL64((a32 ^ d2), 15);
    b4 = ROL64((a13 ^ d3), 56);
    b0 = ROL64((a44 ^ d4), 27);
    a20 = b0 ^ ((~b1) & b2);
    a01 = b1 ^ ((~b2) & b3);
    a32 = b2 ^ ((~b3) & b4);
    a13 = b3 ^ ((~b4) & b0);
    a44 = b4 ^ ((~b0) & b1);

    b3 = ROL64((a10 ^ d0), 41);
    b4 = ROL64((a41 ^ d1), 2);
    b0 = ROL64((a22 ^ d2), 62);
    b1 = ROL64((a03 ^ d3), 55);
    b2 = ROL64((a34 ^ d4), 39);
    a10 = b0 ^ ((~b1) & b2);
    a41 = b1 ^ ((~b2) & b3);
    a22 = b2 ^ ((~b3) & b4);
    a03 = b3 ^ ((~b4) & b0);
    a34 = b4 ^ ((~b0) & b1);

    c0 = a00 ^ a40 ^ a30 ^ a20 ^ a10;
    c1 = a31 ^ a21 ^ a11 ^ a01 ^ a41;
    c2 = a12 ^ a02 ^ a42 ^ a32 ^ a22;
    c3 = a43 ^ a33 ^ a23 ^ a13 ^ a03;
    c4 = a24 ^ a14 ^ a04 ^ a44 ^ a34;
    d0 = c4 ^ ROL64(c1, 1);
    d1 = c0 ^ ROL64(c2, 1);
    d2 = c1 ^ ROL64(c3, 1);
    d3 = c2 ^ ROL64(c4, 1);
    d4 = c3 ^ ROL64(c0, 1);

    b0 = (a00 ^ d0);
    b1 = ROL64((a21 ^ d1), 44);
    b2 = ROL64((a42 ^ d2), 43);
    b3 = ROL64((a13 ^ d3), 21);
    b4 = ROL64((a34 ^ d4), 14);
    a00 = b0 ^ ((~b1) & b2);
    a00 ^= RC[i + 2];
    a21 = b1 ^ ((~b2) & b3);
    a42 = b2 ^ ((~b3) & b4);
    a13 = b3 ^ ((~b4) & b0);
    a34 = b4 ^ ((~b0) & b1);

    b2 = ROL64((a30 ^ d0), 3);
    b3 = ROL64((a01 ^ d1), 45);
    b4 = ROL64((a22 ^ d2), 61);
    b0 = ROL64((a43 ^ d3), 28);
    b1 = ROL64((a14 ^ d4), 20);
    a30 = b0 ^ ((~b1) & b2);
    a01 = b1 ^ ((~b2) & b3);
    a22 = b2 ^ ((~b3) & b4);
    a43 = b3 ^ ((~b4) & b0);
    a14 = b4 ^ ((~b0) & b1);

    b4 = ROL64((a10 ^ d0), 18);
    b0 = ROL64((a31 ^ d1), 1);
    b1 = ROL64((a02 ^ d2), 6);
    b2 = ROL64((a23 ^ d3), 25);
    b3 = ROL64((a44 ^ d4), 8);
    a10 = b0 ^ ((~b1) & b2);
    a31 = b1 ^ ((~b2) & b3);
    a02 = b2 ^ ((~b3) & b4);
    a23 = b3 ^ ((~b4) & b0);
    a44 = b4 ^ ((~b0) & b1);

    b1 = ROL64((a40 ^ d0), 36);
    b2 = ROL64((a11 ^ d1), 10);
    b3 = ROL64((a32 ^ d2), 15);
    b4 = ROL64((a03 ^ d3), 56);
    b0 = ROL64((a24 ^ d4), 27);
    a40 = b0 ^ ((~b1) & b2);
    a11 = b1 ^ ((~b2) & b3);
    a32 = b2 ^ ((~b3) & b4);
    a03 = b3 ^ ((~b4) & b0);
    a24 = b4 ^ ((~b0) & b1);

    b3 = ROL64((a20 ^ d0), 41);
    b4 = ROL64((a41 ^ d1), 2);
    b0 = ROL64((a12 ^ d2), 62);
    b1 = ROL64((a33 ^ d3), 55);
    b2 = ROL64((a04 ^ d4), 39);
    a20 = b0 ^ ((~b1) & b2);
    a41 = b1 ^ ((~b2) & b3);
    a12 = b2 ^ ((~b3) & b4);
    a33 = b3 ^ ((~b4) & b0);
    a04 = b4 ^ ((~b0) & b1);

    c0 = a00 ^ a30 ^ a10 ^ a40 ^ a20;
    c1 = a21 ^ a01 ^ a31 ^ a11 ^ a41;
    c2 = a42 ^ a22 ^ a02 ^ a32 ^ a12;
    c3 = a13 ^ a43 ^ a23 ^ a03 ^ a33;
    c4 = a34 ^ a14 ^ a44 ^ a24 ^ a04;
    d0 = c4 ^ ROL64(c1, 1);
    d1 = c0 ^ ROL64(c2, 1);
    d2 = c1 ^ ROL64(c3, 1);
    d3 = c2 ^ ROL64(c4, 1);
    d4 = c3 ^ ROL64(c0, 1);

    b0 = (a00 ^ d0);
    b1 = ROL64((a01 ^ d1), 44);
    b2 = ROL64((a02 ^ d2), 43);
    b3 = ROL64((a03 ^ d3), 21);
    b4 = ROL64((a04 ^ d4), 14);
    a00 = b0 ^ ((~b1) & b2);
    a00 ^= RC[i + 3];
    a01 = b1 ^ ((~b2) & b3);
    a02 = b2 ^ ((~b3) & b4);
    a03 = b3 ^ ((~b4) & b0);
    a04 = b4 ^ ((~b0) & b1);

    b2 = ROL64((a10 ^ d0), 3);
    b3 = ROL64((a11 ^ d1), 45);
    b4 = ROL64((a12 ^ d2), 61);
    b0 = ROL64((a13 ^ d3), 28);
    b1 = ROL64((a14 ^ d4), 20);
    a10 = b0 ^ ((~b1) & b2);
    a11 = b1 ^ ((~b2) & b3);
    a12 = b2 ^ ((~b3) & b4);
    a13 = b3 ^ ((~b4) & b0);
    a14 = b4 ^ ((~b0) & b1);

    b4 = ROL64((a20 ^ d0), 18);
    b0 = ROL64((a21 ^ d1), 1);
    b1 = ROL64((a22 ^ d2), 6);
    b2 = ROL64((a23 ^ d3), 25);
    b3 = ROL64((a24 ^ d4), 8);
    a20 = b0 ^ ((~b1) & b2);
    a21 = b1 ^ ((~b2) & b3);
    a22 = b2 ^ ((~b3) & b4);
    a23 = b3 ^ ((~b4) & b0);
    a24 = b4 ^ ((~b0) & b1);

    b1 = ROL64((a30 ^ d0), 36);
    b2 = ROL64((a31 ^ d1), 10);
    b3 = ROL64((a32 ^ d2), 15);
    b4 = ROL64((a33 ^ d3), 56);
    b0 = ROL64((a34 ^ d4), 27);
    a30 = b0 ^ ((~b1) & b2);
    a31 = b1 ^ ((~b2) & b3);
    a32 = b2 ^ ((~b3) & b4);
    a33 = b3 ^ ((~b4) & b0);
    a34 = b4 ^ ((~b0) & b1);

    b3 = ROL64((a40 ^ d0), 41);
    b4 = ROL64((a41 ^ d1), 2);
    b0 = ROL64((a42 ^ d2), 62);
    b1 = ROL64((a43 ^ d3), 55);
    b2 = ROL64((a44 ^ d4), 39);
    a40 = b0 ^ ((~b1) & b2);
    a41 = b1 ^ ((~b2) & b3);
    a42 = b2 ^ ((~b3) & b4);
    a43 = b3 ^ ((~b4) & b0);
    a44 = b4 ^ ((~b0) & b1);
  }
}

/*
** Initialize a new hash.  iSize determines the size of the hash
** in bits and should be one of 224, 256, 384, or 512.  Or iSize
** can be zero to use the default hash size of 256 bits.
*/
static void SHA3Init(SHA3Context *p, int iSize) {
  memset(p, 0, sizeof(*p));
  if (iSize >= 128 && iSize <= 512) {
    p->nRate = (1600 - ((iSize + 31) & ~31) * 2) / 8;
  } else {
    p->nRate = (1600 - 2 * 256) / 8;
  }
#if SHA3_BYTEORDER == 1234
  /* Known to be little-endian at compile-time. No-op */
#elif SHA3_BYTEORDER == 4321
  p->ixMask = 7; /* Big-endian */
#else
  {
    static unsigned int one = 1;
    if (1 == *(unsigned char *)&one) {
      /* Little endian.  No byte swapping. */
      p->ixMask = 0;
    } else {
      /* Big endian.  Byte swap. */
      p->ixMask = 7;
    }
  }
#endif
}

/*
** Make consecutive calls to the SHA3Update function to add new content
** to the hash
*/
static void SHA3Update(SHA3Context *p, const unsigned char *aData,
                       unsigned int nData) {
  unsigned int i = 0;
#if SHA3_BYTEORDER == 1234
  if ((p->nLoaded % 8) == 0 && ((aData - (const unsigned char *)0) & 7) == 0) {
    for (; i + 7 < nData; i += 8) {
      p->u.s[p->nLoaded / 8] ^= *(u64 *)&aData[i];
      p->nLoaded += 8;
      if (p->nLoaded >= p->nRate) {
        KeccakF1600Step(p);
        p->nLoaded = 0;
      }
    }
  }
#endif
  for (; i < nData; i++) {
#if SHA3_BYTEORDER == 1234
    p->u.x[p->nLoaded] ^= aData[i];
#elif SHA3_BYTEORDER == 4321
    p->u.x[p->nLoaded ^ 0x07] ^= aData[i];
#else
    p->u.x[p->nLoaded ^ p->ixMask] ^= aData[i];
#endif
    p->nLoaded++;
    if (p->nLoaded == p->nRate) {
      KeccakF1600Step(p);
      p->nLoaded = 0;
    }
  }
}

/*
** After all content has been added, invoke SHA3Final() to compute
** the final hash.  The function returns a pointer to the binary
** hash value.
*/
static unsigned char *SHA3Final(SHA3Context *p) {
  unsigned int i;
  if (p->nLoaded == p->nRate - 1) {
    const unsigned char c1 = 0x86;
    SHA3Update(p, &c1, 1);
  } else {
    const unsigned char c2 = 0x06;
    const unsigned char c3 = 0x80;
    SHA3Update(p, &c2, 1);
    p->nLoaded = p->nRate - 1;
    SHA3Update(p, &c3, 1);
  }
  for (i = 0; i < p->nRate; i++) {
    p->u.x[i + p->nRate] = p->u.x[i ^ p->ixMask];
  }
  return &p->u.x[p->nRate];
}
/* End of the hashing logic
*****************************************************************************/

/*
** Implementation of the sha3(X,SIZE) function.
**
** Return a BLOB which is the SIZE-bit SHA3 hash of X.  The default
** size is 256.  If X is a BLOB, it is hashed as is.
** For all other non-NULL types of input, X is converted into a UTF-8 string
** and the string is hashed without the trailing 0x00 terminator.  The hash
** of a NULL value is NULL.
*/
static void sha3Func(sqlite3_context *context, int argc, sqlite3_value **argv) {
  SHA3Context cx;
  int eType = sqlite3_value_type(argv[0]);
  int nByte = sqlite3_value_bytes(argv[0]);
  int iSize;
  if (argc == 1) {
    iSize = 256;
  } else {
    iSize = sqlite3_value_int(argv[1]);
    if (iSize != 224 && iSize != 256 && iSize != 384 && iSize != 512) {
      sqlite3_result_error(context,
                           "SHA3 size should be one of: 224 256 "
                           "384 512",
                           -1);
      return;
    }
  }
  if (eType == SQLITE_NULL)
    return;
  SHA3Init(&cx, iSize);
  if (eType == SQLITE_BLOB) {
    SHA3Update(&cx, sqlite3_value_blob(argv[0]), nByte);
  } else {
    SHA3Update(&cx, sqlite3_value_text(argv[0]), nByte);
  }

  int sha3len = iSize / 8;
  char *hex = to_hex(SHA3Final(&cx), sha3len);
  if (!hex) {
    sqlite3_result_error_nomem(context);
    return;
  }
  sqlite3_result_text(context, hex, sha3len * 2, sqlite3_free);

  //  sqlite3_result_blob(context, SHA3Final(&cx), iSize/8, SQLITE_TRANSIENT);
}

// end of code

static void bf_create_digest(sqlite3_context *ctx,
                             int nargs __attribute__((unused)),
                             sqlite3_value **args) {
  if (sqlite3_value_type(args[0]) == SQLITE_NULL ||
      sqlite3_value_type(args[1]) == SQLITE_NULL) {
    return;
  }
  const EVP_MD *algo =
      EVP_get_digestbyname((const char *)sqlite3_value_text(args[0]));
  if (!algo) {
    return;
  }
  const unsigned char *data = sqlite3_value_blob(args[1]);
  if (!data) {
    return;
  }
  int datalen = sqlite3_value_bytes(args[1]);

  unsigned char md[EVP_MAX_MD_SIZE];
  unsigned int mdlen;
  EVP_MD_CTX *hashctx;

#ifdef HAVE_EVP_MD_CTX_NEW
  hashctx = EVP_MD_CTX_new();
#else
  hashctx = EVP_MD_CTX_create();
#endif
  if (!hashctx) {
    sqlite3_result_error_nomem(ctx);
    return;
  }
  if (!EVP_DigestInit_ex(hashctx, algo, NULL)) {
    sqlite3_result_error(ctx, "EVP_DigestInit_ex failed", -1);
#ifdef HAVE_EVP_MD_CTX_NEW
    EVP_MD_CTX_free(hashctx);
#else
    EVP_MD_CTX_destroy(hashctx);
#endif
    return;
  }
  if (!EVP_DigestUpdate(hashctx, data, datalen)) {
    sqlite3_result_error(ctx, "EVP_DigestUpdate failed", -1);
#ifdef HAVE_EVP_MD_CTX_NEW
    EVP_MD_CTX_free(hashctx);
#else
    EVP_MD_CTX_destroy(hashctx);
#endif
    return;
  }
  if (!EVP_DigestFinal_ex(hashctx, md, &mdlen)) {
    sqlite3_result_error(ctx, "EVP_DigestFinal_ex failed", -1);
#ifdef HAVE_EVP_MD_CTX_NEW
    EVP_MD_CTX_free(hashctx);
#else
    EVP_MD_CTX_destroy(hashctx);
#endif
    return;
  }
#ifdef HAVE_EVP_MD_CTX_NEW
  EVP_MD_CTX_free(hashctx);
#else
  EVP_MD_CTX_destroy(hashctx);
#endif

  sqlite3_result_blob(ctx, md, mdlen, SQLITE_TRANSIENT);
}

static void bf_hmac(sqlite3_context *ctx, int nargs __attribute__((unused)),
                    sqlite3_value **args) {
  if (sqlite3_value_type(args[0]) == SQLITE_NULL ||
      sqlite3_value_type(args[1]) == SQLITE_NULL ||
      sqlite3_value_type(args[2]) == SQLITE_NULL) {
    return;
  }

  const EVP_MD *algo =
      EVP_get_digestbyname((const char *)sqlite3_value_text(args[0]));
  if (!algo) {
    return;
  }

  const unsigned char *secret = sqlite3_value_blob(args[1]);
  if (!secret) {
    return;
  }
  int slen = sqlite3_value_bytes(args[1]);

  const unsigned char *data = sqlite3_value_blob(args[2]);
  if (!data) {
    return;
  }
  int dlen = sqlite3_value_bytes(args[2]);

  unsigned char md[EVP_MAX_MD_SIZE];
  unsigned int md_len;

  if (!HMAC(algo, secret, slen, data, dlen, md, &md_len)) {
    sqlite3_result_error(ctx, "HMAC failed", -1);
    return;
  }
  char *hex = to_hex(md, md_len);
  if (!hex) {
    sqlite3_result_error_nomem(ctx);
    return;
  }
  sqlite3_result_text(ctx, hex, md_len * 2, sqlite3_free);
}

// make a key from a 16 byte or greater blob. key must point to 16
// bytes of space.
static void make_aes_key(unsigned char *restrict key,
                         const unsigned char *restrict raw, int len) {
  assert(len >= 16);
  memset(key, '\0', 16);
  for (int i = 0, j = 0; i < len; i += 1, j = ((j + 1) % 16)) {
    key[j] ^= raw[i];
  }
}

static void bf_aes_encrypt(sqlite3_context *ctx,
                           int nargs __attribute__((unused)),
                           sqlite3_value **args) {
  if (sqlite3_value_type(args[0]) == SQLITE_NULL ||
      sqlite3_value_type(args[1]) == SQLITE_NULL) {
    return;
  }

  const EVP_CIPHER *cipher = EVP_aes_128_ecb();

  const unsigned char *rawkey = sqlite3_value_blob(args[1]);
  if (!rawkey) {
    return;
  }
  int rawkeylen = sqlite3_value_bytes(args[1]);

  if (rawkeylen < EVP_CIPHER_key_length(cipher)) {
    sqlite3_result_error(ctx, "invalid key size", -1);
    return;
  }

  const unsigned char *data = sqlite3_value_blob(args[0]);
  if (!data) {
    return;
  }
  int datalen = sqlite3_value_bytes(args[0]);

  EVP_CIPHER_CTX *aesctx = EVP_CIPHER_CTX_new();
  if (!aesctx) {
    sqlite3_result_error_nomem(ctx);
    return;
  }

  unsigned char key[16];
  make_aes_key(key, rawkey, rawkeylen);

  if (!EVP_EncryptInit_ex(aesctx, cipher, NULL, key, NULL)) {
    sqlite3_result_error(ctx, "EVP_EncryptInit_ex failed", -1);
    EVP_CIPHER_CTX_free(aesctx);
    return;
  }

  int blocklen = EVP_CIPHER_block_size(cipher);
  int aes_len = datalen + blocklen * 2;
  unsigned char *aes = sqlite3_malloc(aes_len);
  if (!aes) {
    sqlite3_result_error_nomem(ctx);
    EVP_CIPHER_CTX_free(aesctx);
    return;
  }
  int written = aes_len - blocklen;
  EVP_CIPHER_CTX_set_padding(aesctx, 1);
  if (!EVP_EncryptUpdate(aesctx, aes, &written, data, datalen)) {
    sqlite3_result_error(ctx, "EVP_EncryptUpdate failed", -1);
    sqlite3_free(aes);
    EVP_CIPHER_CTX_free(aesctx);
    return;
  }

  aes_len = written;
  written = blocklen;
  if (!EVP_EncryptFinal_ex(aesctx, aes + aes_len, &written)) {
    sqlite3_result_error(ctx, "EVP_EncryptFinal_ex failed", -1);
    sqlite3_free(aes);
    EVP_CIPHER_CTX_free(aesctx);
    return;
  }
  aes_len += written;

  sqlite3_result_blob(ctx, aes, aes_len, sqlite3_free);
}

static void bf_aes_decrypt(sqlite3_context *ctx,
                           int nargs __attribute__((unused)),
                           sqlite3_value **args) {
  if (sqlite3_value_type(args[0]) == SQLITE_NULL ||
      sqlite3_value_type(args[1]) == SQLITE_NULL) {
    return;
  }

  const EVP_CIPHER *cipher = EVP_aes_128_ecb();

  const unsigned char *rawkey = sqlite3_value_blob(args[1]);
  if (!rawkey) {
    return;
  }
  int rawkeylen = sqlite3_value_bytes(args[1]);

  if (rawkeylen < EVP_CIPHER_key_length(cipher)) {
    sqlite3_result_error(ctx, "invalid key size", -1);
    return;
  }

  const unsigned char *data = sqlite3_value_blob(args[0]);
  if (!data) {
    return;
  }
  int datalen = sqlite3_value_bytes(args[0]);

  EVP_CIPHER_CTX *aesctx = EVP_CIPHER_CTX_new();
  if (!aesctx) {
    sqlite3_result_error_nomem(ctx);
    return;
  }

  unsigned char key[16];
  make_aes_key(key, rawkey, rawkeylen);

  if (!EVP_DecryptInit_ex(aesctx, cipher, NULL, key, NULL)) {
    sqlite3_result_error(ctx, "EVP_DecryptInit_ex failed", -1);
    EVP_CIPHER_CTX_free(aesctx);
    return;
  }

  int blocklen = EVP_CIPHER_block_size(cipher);
  int plain_len = datalen + blocklen * 2;
  unsigned char *plain = sqlite3_malloc(plain_len);
  if (!plain) {
    sqlite3_result_error_nomem(ctx);
    EVP_CIPHER_CTX_free(aesctx);
    return;
  }
  int written = plain_len - blocklen;
  EVP_CIPHER_CTX_set_padding(aesctx, 1);
  if (!EVP_DecryptUpdate(aesctx, plain, &written, data, datalen)) {
    // sqlite3_result_error(ctx, "EVP_DecryptUpdate failed", -1);
    sqlite3_free(plain);
    EVP_CIPHER_CTX_free(aesctx);
    return;
  }

  plain_len = written;
  written = blocklen;
  if (!EVP_DecryptFinal_ex(aesctx, plain + plain_len, &written)) {
    // sqlite3_result_error(ctx, "EVP_DecryptFinal_ex failed", -1);
    sqlite3_free(plain);
    EVP_CIPHER_CTX_free(aesctx);
    return;
  }
  plain_len += written;

  sqlite3_result_blob(ctx, plain, plain_len, sqlite3_free);
}

#if defined(__GNUC__) && (defined(__x86_64) || defined(__i386))
/* SSE 4.2 function to check to see if a string is just hex digits. 3-4
   times faster than the plain version. */
__attribute__((target("sse4.2"))) static void
bf_unhex_sse42(sqlite3_context *ctx, int nargs __attribute__((unused)),
               sqlite3_value **args) {
  static const signed char ___m128i_shift_right[31] = {
      0,  1,  2,  3,  4,  5,  6,  7,  8,  9,  10, 11, 12, 13, 14, 15,
      -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1};
  static unsigned char mapping[UCHAR_MAX + 1] = {
      ['0'] = 0,  ['1'] = 1,  ['2'] = 2,  ['3'] = 3,  ['4'] = 4,  ['5'] = 5,
      ['6'] = 6,  ['7'] = 7,  ['8'] = 8,  ['9'] = 9,  ['A'] = 10, ['B'] = 11,
      ['C'] = 12, ['D'] = 13, ['E'] = 14, ['F'] = 15, ['a'] = 10, ['b'] = 10,
      ['c'] = 12, ['d'] = 13, ['e'] = 14, ['f'] = 15};
#define CMP_FLAGS                                                              \
  _SIDD_UBYTE_OPS | _SIDD_CMP_RANGES | _SIDD_LEAST_SIGNIFICANT |               \
      _SIDD_NEGATIVE_POLARITY
  __m128i hexmask =
      _mm_setr_epi8('0', '9', 'A', 'F', 'a', 'f', 0, 0, 0, 0, 0, 0, 0, 0, 0, 0);

  if (sqlite3_value_type(args[0]) == SQLITE_NULL) {
    return;
  }
  const unsigned char *hex = sqlite3_value_text(args[0]);
  if (!hex) {
    return;
  }
  int hexlen = sqlite3_value_bytes(args[0]);

  if (hexlen & 1) {
    return;
  }

  /* Get hex 16-byte aligned to avoid page boundary issues */
  int offset = ((size_t)hex & 15);
  if (offset) {
    hex = (const unsigned char *)((size_t)hex & -16);
    __m128i chars = _mm_load_si128((__m128i *)hex);
    chars = _mm_shuffle_epi8(
        chars, _mm_loadu_si128((__m128i *)(___m128i_shift_right + offset)));
    int badchar = _mm_cmpistri(hexmask, chars, CMP_FLAGS);
    if (badchar < 16 - offset) {
      if (hex[offset + badchar] == '\0') {
        goto cnvt;
      } else {
        return;
      }
    }
    hex += 16;
  }

  while (1) {
    __m128i chars = _mm_load_si128((__m128i *)hex);
    int badchar = _mm_cmpistri(hexmask, chars, CMP_FLAGS);
    if (badchar < 16) {
      if (hex[badchar] == '\0') {
        goto cnvt;
      } else {
        return;
      }
    }
    hex += 16;
  }

cnvt:
  (void)1;
  unsigned char *blob = sqlite3_malloc(hexlen / 2);
  if (!blob) {
    sqlite3_result_error_nomem(ctx);
    return;
  }

  /* TODO: Merge this into the above loop. Need to figure out
     how to handle an encoded byte that spans two 16-byte chunks. */
  hex = sqlite3_value_text(args[0]);
  for (int i = 0, j = 0; i < hexlen; i += 2, j += 1) {
    blob[j] = (mapping[i] << 4) | mapping[i + 1];
  }
  sqlite3_result_blob(ctx, blob, hexlen / 2, sqlite3_free);
#undef CMP_FLAGS
}
#endif

static void bf_unhex_sw(sqlite3_context *ctx, int nargs __attribute__((unused)),
                        sqlite3_value **args) {
  static unsigned char mapping[UCHAR_MAX + 1] = {
      ['0'] = 1,  ['1'] = 2,  ['2'] = 3,  ['3'] = 4,  ['4'] = 5,  ['5'] = 6,
      ['6'] = 7,  ['7'] = 8,  ['8'] = 9,  ['9'] = 10, ['A'] = 11, ['B'] = 12,
      ['C'] = 13, ['D'] = 14, ['E'] = 15, ['F'] = 16, ['a'] = 11, ['b'] = 11,
      ['c'] = 13, ['d'] = 14, ['e'] = 15, ['f'] = 16};

  const unsigned char *hex = sqlite3_value_text(args[0]);
  if (!hex) {
    return;
  }
  int hexlen = sqlite3_value_bytes(args[0]);

  if (hexlen & 1) {
    return;
  }

  unsigned char *blob = sqlite3_malloc(hexlen / 2);
  if (!blob) {
    sqlite3_result_error_nomem(ctx);
    return;
  }

  for (int i = 0, j = 0; i < hexlen; i += 2, j += 1) {
    int high = mapping[hex[i]];
    int low = mapping[hex[i + 1]];

    if (!high || !low) {
      sqlite3_free(blob);
      return;
    }

    blob[j] = ((high - 1) << 4) | (low - 1);
  }

  sqlite3_result_blob(ctx, blob, hexlen / 2, sqlite3_free);
}

static void bf_to_base64(sqlite3_context *ctx, int narg __attribute__((unused)),
                         sqlite3_value **args) {
  const unsigned char *blob = sqlite3_value_blob(args[0]);
  if (!blob) {
    return;
  }
  int bloblen = sqlite3_value_bytes(args[0]);

  BIO *bio, *b64, *bmem;
  b64 = BIO_new(BIO_f_base64());
  if (!b64) {
    sqlite3_result_error_nomem(ctx);
    return;
  }
  bmem = BIO_new(BIO_s_mem());
  if (!bmem) {
    sqlite3_result_error_nomem(ctx);
    BIO_free(b64);
    return;
  }
  bio = BIO_push(b64, bmem);
  if (BIO_write(bio, blob, bloblen) != bloblen) {
    sqlite3_result_error(ctx, "BIO_write failed", -1);
    BIO_free_all(bio);
    return;
  }

  BIO_flush(bio);
  char *membuf;
  int len = BIO_get_mem_data(bmem, &membuf);
  sqlite3_result_text(ctx, membuf, len, SQLITE_TRANSIENT);
  BIO_free_all(bio);
}

static void bf_from_base64(sqlite3_context *ctx,
                           int narg __attribute__((unused)),
                           sqlite3_value **args) {
  const unsigned char *encoded = sqlite3_value_text(args[0]);
  if (!encoded) {
    return;
  }
  int enclen = sqlite3_value_bytes(args[0]);

  BIO *bio, *b64, *bmem;
  b64 = BIO_new(BIO_f_base64());
  if (!b64) {
    sqlite3_result_error_nomem(ctx);
    return;
  }
  bmem = BIO_new_mem_buf(encoded, enclen);
  if (!bmem) {
    sqlite3_result_error_nomem(ctx);
    BIO_free(b64);
    return;
  }
  bio = BIO_push(b64, bmem);

  unsigned char buffer[8192];
  int r;
  // Unfortunately, the sqlite3_str_XXX() API is too new to use
  unsigned char *result = NULL;
  int totlen = 0;
  while ((r = BIO_read(bio, buffer, sizeof buffer)) > 0) {
    unsigned char *newresult = sqlite3_realloc(result, totlen + r);
    if (!newresult) {
      sqlite3_result_error_nomem(ctx);
      sqlite3_free(result);
      BIO_free_all(bio);
      return;
    }
    result = newresult;
    memcpy(result + totlen, buffer, r);
    totlen += r;
  }
  BIO_free_all(bio);

  if (r < 0) {
    sqlite3_free(result);
    return;
  }

  sqlite3_result_blob(ctx, result, totlen, sqlite3_free);
}

#ifdef ZLIB_FOUND
static void bf_crc32(sqlite3_context *ctx, int nargs __attribute__((unused)),
                     sqlite3_value **args) {
  if (sqlite3_value_type(args[0]) == SQLITE_NULL) {
    return;
  }
  const void *blob = sqlite3_value_blob(args[0]);
  if (!blob) {
    return;
  }
  size_t blen = sqlite3_value_bytes(args[0]);
  sqlite3_result_int64(ctx, crc32(0L, blob, blen));
}
#endif

#if defined(__GNUC__) && (defined(__x86_64) || defined(__i386)) &&             \
    defined(CMAKE_USE_PTHREADS_INIT)
#define HAVE_CRC32C
/* Consider rewriting to use intrinsic functions instead of inline
 * assembly?
 */
/* crc32c.c -- compute CRC-32C using the Intel crc32 instruction
 * Copyright (C) 2013 Mark Adler
 * Version 1.1  1 Aug 2013  Mark Adler
 */

/*
  This software is provided 'as-is', without any express or implied
  warranty.  In no event will the author be held liable for any damages
  arising from the use of this software.

  Permission is granted to anyone to use this software for any purpose,
  including commercial applications, and to alter it and redistribute it
  freely, subject to the following restrictions:

  1. The origin of this software must not be misrepresented; you must not
     claim that you wrote the original software. If you use this software
     in a product, an acknowledgment in the product documentation would be
     appreciated but is not required.
  2. Altered source versions must be plainly marked as such, and must not be
     misrepresented as being the original software.
  3. This notice may not be removed or altered from any source distribution.

  Mark Adler
  madler@alumni.caltech.edu
 */

/* Use hardware CRC instruction on Intel SSE 4.2 processors.  This computes a
   CRC-32C, *not* the CRC-32 used by Ethernet and zip, gzip, etc.  A software
   version is provided as a fall-back, as well as for speed comparisons. */

/* Version history:
   1.0  10 Feb 2013  First version
   1.1   1 Aug 2013  Correct comments on why three crc instructions in parallel
 */

#include <pthread.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

/* CRC-32C (iSCSI) polynomial in reversed bit order. */
#define POLY 0x82f63b78

/* Table for a quadword-at-a-time software crc. */
static pthread_once_t crc32c_once_sw = PTHREAD_ONCE_INIT;
static uint32_t crc32c_table[8][256];

/* Construct table for software CRC-32C calculation. */
static void crc32c_init_sw(void) {
  uint32_t n, crc, k;

  for (n = 0; n < 256; n++) {
    crc = n;
    crc = crc & 1 ? (crc >> 1) ^ POLY : crc >> 1;
    crc = crc & 1 ? (crc >> 1) ^ POLY : crc >> 1;
    crc = crc & 1 ? (crc >> 1) ^ POLY : crc >> 1;
    crc = crc & 1 ? (crc >> 1) ^ POLY : crc >> 1;
    crc = crc & 1 ? (crc >> 1) ^ POLY : crc >> 1;
    crc = crc & 1 ? (crc >> 1) ^ POLY : crc >> 1;
    crc = crc & 1 ? (crc >> 1) ^ POLY : crc >> 1;
    crc = crc & 1 ? (crc >> 1) ^ POLY : crc >> 1;
    crc32c_table[0][n] = crc;
  }
  for (n = 0; n < 256; n++) {
    crc = crc32c_table[0][n];
    for (k = 1; k < 8; k++) {
      crc = crc32c_table[0][crc & 0xff] ^ (crc >> 8);
      crc32c_table[k][n] = crc;
    }
  }
}

/* Table-driven software version as a fall-back.  This is about 15 times slower
   than using the hardware instructions.  This assumes little-endian integers,
   as is the case on Intel processors that the assembler code here is for. */
static uint32_t crc32c_sw(uint32_t crci, const void *buf, size_t len) {
  const unsigned char *next = buf;
  uint64_t crc;

  pthread_once(&crc32c_once_sw, crc32c_init_sw);
  crc = crci ^ 0xffffffff;
  while (len && ((uintptr_t)next & 7) != 0) {
    crc = crc32c_table[0][(crc ^ *next++) & 0xff] ^ (crc >> 8);
    len--;
  }
  while (len >= 8) {
    crc ^= *(uint64_t *)next;
    crc = crc32c_table[7][crc & 0xff] ^ crc32c_table[6][(crc >> 8) & 0xff] ^
          crc32c_table[5][(crc >> 16) & 0xff] ^
          crc32c_table[4][(crc >> 24) & 0xff] ^
          crc32c_table[3][(crc >> 32) & 0xff] ^
          crc32c_table[2][(crc >> 40) & 0xff] ^
          crc32c_table[1][(crc >> 48) & 0xff] ^ crc32c_table[0][crc >> 56];
    next += 8;
    len -= 8;
  }
  while (len) {
    crc = crc32c_table[0][(crc ^ *next++) & 0xff] ^ (crc >> 8);
    len--;
  }
  return (uint32_t)crc ^ 0xffffffff;
}

/* Multiply a matrix times a vector over the Galois field of two elements,
   GF(2).  Each element is a bit in an unsigned integer.  mat must have at
   least as many entries as the power of two for most significant one bit in
   vec. */
static inline uint32_t gf2_matrix_times(uint32_t *mat, uint32_t vec) {
  uint32_t sum;

  sum = 0;
  while (vec) {
    if (vec & 1)
      sum ^= *mat;
    vec >>= 1;
    mat++;
  }
  return sum;
}

/* Multiply a matrix by itself over GF(2).  Both mat and square must have 32
   rows. */
static inline void gf2_matrix_square(uint32_t *square, uint32_t *mat) {
  int n;

  for (n = 0; n < 32; n++)
    square[n] = gf2_matrix_times(mat, mat[n]);
}

/* Construct an operator to apply len zeros to a crc.  len must be a power of
   two.  If len is not a power of two, then the result is the same as for the
   largest power of two less than len.  The result for len == 0 is the same as
   for len == 1.  A version of this routine could be easily written for any
   len, but that is not needed for this application. */
static void crc32c_zeros_op(uint32_t *even, size_t len) {
  int n;
  uint32_t row;
  uint32_t odd[32]; /* odd-power-of-two zeros operator */

  /* put operator for one zero bit in odd */
  odd[0] = POLY; /* CRC-32C polynomial */
  row = 1;
  for (n = 1; n < 32; n++) {
    odd[n] = row;
    row <<= 1;
  }

  /* put operator for two zero bits in even */
  gf2_matrix_square(even, odd);

  /* put operator for four zero bits in odd */
  gf2_matrix_square(odd, even);

  /* first square will put the operator for one zero byte (eight zero bits),
     in even -- next square puts operator for two zero bytes in odd, and so
     on, until len has been rotated down to zero */
  do {
    gf2_matrix_square(even, odd);
    len >>= 1;
    if (len == 0)
      return;
    gf2_matrix_square(odd, even);
    len >>= 1;
  } while (len);

  /* answer ended up in odd -- copy to even */
  for (n = 0; n < 32; n++)
    even[n] = odd[n];
}

/* Take a length and build four lookup tables for applying the zeros operator
   for that length, byte-by-byte on the operand. */
static void crc32c_zeros(uint32_t zeros[][256], size_t len) {
  uint32_t n;
  uint32_t op[32];

  crc32c_zeros_op(op, len);
  for (n = 0; n < 256; n++) {
    zeros[0][n] = gf2_matrix_times(op, n);
    zeros[1][n] = gf2_matrix_times(op, n << 8);
    zeros[2][n] = gf2_matrix_times(op, n << 16);
    zeros[3][n] = gf2_matrix_times(op, n << 24);
  }
}

/* Apply the zeros operator table to crc. */
static inline uint32_t crc32c_shift(uint32_t zeros[][256], uint32_t crc) {
  return zeros[0][crc & 0xff] ^ zeros[1][(crc >> 8) & 0xff] ^
         zeros[2][(crc >> 16) & 0xff] ^ zeros[3][crc >> 24];
}

/* Block sizes for three-way parallel crc computation.  LONG and SHORT must
   both be powers of two.  The associated string constants must be set
   accordingly, for use in constructing the assembler instructions. */
#define LONG 8192
#define LONGx1 "8192"
#define LONGx2 "16384"
#define SHORT 256
#define SHORTx1 "256"
#define SHORTx2 "512"

/* Tables for hardware crc that shift a crc by LONG and SHORT zeros. */
static pthread_once_t crc32c_once_hw = PTHREAD_ONCE_INIT;
static uint32_t crc32c_long[4][256];
static uint32_t crc32c_short[4][256];

/* Initialize tables for shifting crcs. */
static void crc32c_init_hw(void) {
  crc32c_zeros(crc32c_long, LONG);
  crc32c_zeros(crc32c_short, SHORT);
}

/* Compute CRC-32C using the Intel hardware instruction. */
static uint32_t crc32c_hw(uint32_t crc, const void *buf, size_t len) {
  const unsigned char *next = buf;
  const unsigned char *end;
  uint64_t crc0, crc1, crc2; /* need to be 64 bits for crc32q */

  /* populate shift tables the first time through */
  pthread_once(&crc32c_once_hw, crc32c_init_hw);

  /* pre-process the crc */
  crc0 = crc ^ 0xffffffff;

  /* compute the crc for up to seven leading bytes to bring the data pointer
     to an eight-byte boundary */
  while (len && ((uintptr_t)next & 7) != 0) {
    __asm__("crc32b\t"
            "(%1), %0"
            : "=r"(crc0)
            : "r"(next), "0"(crc0));
    next++;
    len--;
  }

  /* compute the crc on sets of LONG*3 bytes, executing three independent crc
     instructions, each on LONG bytes -- this is optimized for the Nehalem,
     Westmere, Sandy Bridge, and Ivy Bridge architectures, which have a
     throughput of one crc per cycle, but a latency of three cycles */
  while (len >= LONG * 3) {
    crc1 = 0;
    crc2 = 0;
    end = next + LONG;
    do {
      __asm__("crc32q\t"
              "(%3), %0\n\t"
              "crc32q\t" LONGx1 "(%3), %1\n\t"
              "crc32q\t" LONGx2 "(%3), %2"
              : "=r"(crc0), "=r"(crc1), "=r"(crc2)
              : "r"(next), "0"(crc0), "1"(crc1), "2"(crc2));
      next += 8;
    } while (next < end);
    crc0 = crc32c_shift(crc32c_long, crc0) ^ crc1;
    crc0 = crc32c_shift(crc32c_long, crc0) ^ crc2;
    next += LONG * 2;
    len -= LONG * 3;
  }

  /* do the same thing, but now on SHORT*3 blocks for the remaining data less
     than a LONG*3 block */
  while (len >= SHORT * 3) {
    crc1 = 0;
    crc2 = 0;
    end = next + SHORT;
    do {
      __asm__("crc32q\t"
              "(%3), %0\n\t"
              "crc32q\t" SHORTx1 "(%3), %1\n\t"
              "crc32q\t" SHORTx2 "(%3), %2"
              : "=r"(crc0), "=r"(crc1), "=r"(crc2)
              : "r"(next), "0"(crc0), "1"(crc1), "2"(crc2));
      next += 8;
    } while (next < end);
    crc0 = crc32c_shift(crc32c_short, crc0) ^ crc1;
    crc0 = crc32c_shift(crc32c_short, crc0) ^ crc2;
    next += SHORT * 2;
    len -= SHORT * 3;
  }

  /* compute the crc on the remaining eight-byte units less than a SHORT*3
     block */
  end = next + (len - (len & 7));
  while (next < end) {
    __asm__("crc32q\t"
            "(%1), %0"
            : "=r"(crc0)
            : "r"(next), "0"(crc0));
    next += 8;
  }
  len &= 7;

  /* compute the crc for up to seven trailing bytes */
  while (len) {
    __asm__("crc32b\t"
            "(%1), %0"
            : "=r"(crc0)
            : "r"(next), "0"(crc0));
    next++;
    len--;
  }

  /* return a post-processed crc */
  return (uint32_t)crc0 ^ 0xffffffff;
}

static void bf_crc32c_sse42(sqlite3_context *ctx,
                            int nargs __attribute__((unused)),
                            sqlite3_value **args) {
  if (sqlite3_value_type(args[0]) == SQLITE_NULL) {
    return;
  }
  const void *blob = sqlite3_value_blob(args[0]);
  if (!blob) {
    return;
  }
  int blen = sqlite3_value_bytes(args[0]);
  sqlite3_result_int64(ctx, crc32c_hw(0, blob, blen));
}

static void bf_crc32c_sw(sqlite3_context *ctx,
                         int nargs __attribute__((unused)),
                         sqlite3_value **args) {
  if (sqlite3_value_type(args[0]) == SQLITE_NULL) {
    return;
  }
  const void *blob = sqlite3_value_blob(args[0]);
  if (!blob) {
    return;
  }
  int blen = sqlite3_value_bytes(args[0]);
  sqlite3_result_int64(ctx, crc32c_sw(0, blob, blen));
}

#endif

#define BLOB_UUID 'u'
#define TEXT_UUID 'U'

static void bf_uuid(sqlite3_context *ctx, int nargs __attribute__((unused)),
                    sqlite3_value **args __attribute__((unused))) {
  unsigned char *raw = sqlite3_malloc(16);
  if (!raw) {
    sqlite3_result_error_nomem(ctx);
    return;
  }
  /* Use OpenSSL's cryptographically strong PRNG instead of sqlite3's PRNG */
  if (RAND_bytes(raw, 16) != 1) {
    unsigned long errcode = ERR_get_error();
    char *errmsg =
        sqlite3_mprintf("RAND_bytes: %s", ERR_reason_error_string(errcode));
    if (!errmsg) {
      sqlite3_result_error_nomem(ctx);
    } else {
      sqlite3_result_error(ctx, errmsg, -1);
      sqlite3_free(errmsg);
    }
    sqlite3_free(raw);
    return;
  }
  raw[6] = (raw[6] & 0xF) | 0x40;
  raw[8] = (raw[8] & 0x3F) | 0x80;
  sqlite3_result_blob(ctx, raw, 16, sqlite3_free);
  sqlite3_result_subtype(ctx, BLOB_UUID);
}

/* Check variant and version of a binary uuid */
static _Bool bf_check_uuid_bits(const unsigned char uuid[static 16]) {
  unsigned int vers = (uuid[6] & 0xF0) >> 4;
  unsigned int var = (uuid[8] & 0xE0) >> 5;

  if (vers == 0 || vers > 5) {
    return 0;
  }

  /* Only handle variant 1 uuids */
  if ((var >> 1) != 2) {
    return 0;
  }
  return 1;
}

/* Check variant and version of a textual uuid */
static _Bool bf_check_uuid_bytes(const unsigned char uuid[static 36]) {
  char vers = uuid[14];

  if (vers <= '0' || vers > '5') {
    return 0;
  }

  char var = uuid[19];
  if (isdigit(var)) {
    var = var - '0';
  } else if (isupper(var)) {
    var = var - 'A' + 10;
  } else {
    var = var - 'a' + 10;
  }
  return (var >> 2) == 2;
}

static void bf_bin_to_uuid(sqlite3_context *ctx,
                           int nargs __attribute__((unused)),
                           sqlite3_value **args) {
  const unsigned char *uuid;
  int blen;
  switch (sqlite3_value_type(args[0])) {
  case SQLITE_NULL:
    return;
  case SQLITE_BLOB:
    uuid = sqlite3_value_blob(args[0]);
    blen = sqlite3_value_bytes(args[0]);
    if (sqlite3_value_subtype(args[0]) == BLOB_UUID) {
      break;
    }
    if (blen == 16 && bf_check_uuid_bits(uuid)) {
      break;
    }
    /* FALLTHROUGH */
  default:
    sqlite3_result_error(ctx, "not a UUID blob", -1);
    return;
  }

  char *as_str =
      sqlite3_mprintf("%02x%02x%02x%02x-%02x%02x-%02x%02x-%02x%02x-"
                      "%02x%02x%02x%02x%02x%02x",
                      uuid[0], uuid[1], uuid[2], uuid[3], uuid[4], uuid[5],
                      uuid[6], uuid[7], uuid[8], uuid[9], uuid[10], uuid[11],
                      uuid[12], uuid[13], uuid[14], uuid[15]);
  if (!as_str) {
    sqlite3_result_error_nomem(ctx);
    return;
  }
  sqlite3_result_text(ctx, as_str, -1, sqlite3_free);
  sqlite3_result_subtype(ctx, TEXT_UUID);
}

static void bf_uuid_to_bin(sqlite3_context *ctx,
                           int nargs __attribute__((unused)),
                           sqlite3_value **args) {
  const unsigned char *uuid;
  int ulen;
  switch (sqlite3_value_type(args[0])) {
  case SQLITE_NULL:
    return;
  case SQLITE_TEXT:
    uuid = sqlite3_value_text(args[0]);
    ulen = sqlite3_value_bytes(args[0]);
    if (sqlite3_value_subtype(args[0]) == TEXT_UUID) {
      break;
    }
    if (ulen == 36) {
      break;
    }
    /* FALLTHROUGH */
  default:
    sqlite3_result_error(ctx, "not a UUID string", -1);
    return;
  }

  unsigned char *raw = sqlite3_malloc(16);
  if (!raw) {
    sqlite3_result_error_nomem(ctx);
    return;
  }
  if (sscanf((const char *)uuid,
             "%2hhx%2hhx%2hhx%2hhx-%2hhx%2hhx-%2hhx%2hhx-%2hhx%2hhx-"
             "%2hhx%2hhx%2hhx%2hhx%2hhx%2hhx",
             raw, raw + 1, raw + 2, raw + 3, raw + 4, raw + 5, raw + 6, raw + 7,
             raw + 8, raw + 9, raw + 10, raw + 11, raw + 12, raw + 13, raw + 14,
             raw + 15) != 16 ||
      !bf_check_uuid_bits(raw)) {
    sqlite3_free(raw);
    sqlite3_result_error(ctx, "not a UUID string", -1);
    return;
  }
  sqlite3_result_blob(ctx, raw, 16, sqlite3_free);
  sqlite3_result_subtype(ctx, BLOB_UUID);
}

static void bf_is_uuid(sqlite3_context *ctx, int nargs __attribute__((unused)),
                       sqlite3_value **args) {
  int valid = 0;
  int type = sqlite3_value_type(args[0]);
  if (type == SQLITE_NULL) {
    return;
  }

  int subtype = sqlite3_value_subtype(args[0]);
  if (subtype == TEXT_UUID || subtype == BLOB_UUID) {
    valid = 1;
    goto done;
  }

  if (type == SQLITE_BLOB) {
    const unsigned char *uuid = sqlite3_value_blob(args[0]);
    if (!uuid) {
      return;
    }
    valid = sqlite3_value_bytes(args[0]) == 16 && bf_check_uuid_bits(uuid);
    goto done;
  } else if (type == SQLITE_TEXT) {
    const unsigned char *uuid = sqlite3_value_text(args[0]);
    if (!uuid) {
      return;
    }
    int len = sqlite3_value_bytes(args[0]);
    if (len != 36) {
      goto done;
    }

    static const _Bool dashes[36] = {[8] = 1, [13] = 1, [18] = 1, [23] = 1};
    for (int i = 0; i < len; i += 1) {
      if (dashes[i]) {
        if (uuid[i] != '-') {
          goto done;
        }
      } else if (!isxdigit(uuid[i])) {
        goto done;
      }
    }
    valid = bf_check_uuid_bytes(uuid);
  }
done:
  sqlite3_result_int(ctx, valid);
}

#ifdef ZLIB_FOUND
static void bf_compress(sqlite3_context *ctx, int nargs __attribute__((unused)),
                        sqlite3_value **args) {
  if (sqlite3_value_type(args[0]) == SQLITE_NULL) {
    return;
  }
  const unsigned char *raw = sqlite3_value_blob(args[0]);
  if (!raw) {
    return;
  }
  uint32_t rsize = sqlite3_value_bytes(args[0]);
  uLong csize = compressBound(rsize);
  unsigned char *c = sqlite3_malloc(csize + 4);
  if (!c) {
    sqlite3_result_error_nomem(ctx);
    return;
  }
  int r = compress(c + 4, &csize, raw, rsize);
  if (r == Z_MEM_ERROR) {
    sqlite3_free(c);
    sqlite3_result_error_nomem(ctx);
  } else if (r == Z_BUF_ERROR) {
    // Shouldn't happen
    sqlite3_free(c);
    sqlite3_result_error(ctx, "compression error", -1);
  } else if (r == Z_OK) {
    rsize = htonl(rsize);
    memcpy(c, &rsize, 4);
    sqlite3_result_blob(ctx, c, csize + 4, sqlite3_free);
  }
}

static void bf_uncompress(sqlite3_context *ctx,
                          int nargs __attribute__((unused)),
                          sqlite3_value **args) {
  if (sqlite3_value_type(args[0]) != SQLITE_BLOB) {
    return;
  }
  const unsigned char *c = sqlite3_value_blob(args[0]);
  if (!c) {
    return;
  }
  int csize = sqlite3_value_bytes(args[0]) - 4;
  uint32_t tmp;
  memcpy(&tmp, c, 4);
  uLong rsize = ntohl(tmp);
  unsigned char *raw = sqlite3_malloc(rsize);
  if (!raw) {
    sqlite3_result_error_nomem(ctx);
    return;
  }
  int r = uncompress(raw, &rsize, c + 4, csize);
  if (r == Z_MEM_ERROR) {
    sqlite3_free(raw);
    sqlite3_result_error_nomem(ctx);
  } else if (r == Z_DATA_ERROR) {
    sqlite3_free(raw);
  } else if (r == Z_BUF_ERROR) {
    // Shouldn't happen
    sqlite3_free(raw);
    sqlite3_result_error(ctx, "uncompression error", -1);
  } else if (r == Z_OK) {
    sqlite3_result_blob(ctx, raw, rsize, sqlite3_free);
  }
}
#endif

#ifdef _WIN32
__declspec(export)
#endif
    int sqlite3_blobfuncs_init(sqlite3 *db,
                               char **pzErrMsg __attribute__((unused)),
                               const sqlite3_api_routines *pApi) {
  SQLITE_EXTENSION_INIT2(pApi);
  struct bf_funcs {
    const char *name;
    int nargs;
    void (*fp)(sqlite3_context *, int, sqlite3_value **);
  } func_table[] = {{"md5", 1, bf_md5},
                    {"sha1", 1, bf_sha1},
                    {"sha2", 1, bf_sha2},
                    {"sha2", 2, bf_sha2},
                    {"sha3", 1, sha3Func},
                    {"sha3", 2, sha3Func},
                    {"create_digest", 2, bf_create_digest},
                    {"hmac", 3, bf_hmac},
                    {"aes_encrypt", 2, bf_aes_encrypt},
                    {"aes_decrypt", 2, bf_aes_decrypt},
                    {"to_base64", 1, bf_to_base64},
                    {"from_base64", 1, bf_from_base64},
                    {"uuid", 0, bf_uuid},
                    {"is_uuid", 1, bf_is_uuid},
                    {"bin_to_uuid", 1, bf_bin_to_uuid},
                    {"uuid_to_bin", 1, bf_uuid_to_bin},
#ifdef ZLIB_FOUND
                    {"crc32", 1, bf_crc32},
                    {"compress", 1, bf_compress},
                    {"uncompress", 1, bf_uncompress},
#endif
                    {NULL, -1, NULL}};

#if defined(__GNUC__) && (defined(__x86_64) || defined(__i386))
  if (__builtin_cpu_supports("sse4.2")) {
    int rc;
    rc = sqlite3_create_function(db, "unhex", 1,
                                 SQLITE_UTF8 | SQLITE_DETERMINISTIC, NULL,
                                 bf_unhex_sse42, NULL, NULL);
    if (rc != SQLITE_OK) {
      return rc;
    }
    rc = sqlite3_create_function(db, "crc32c", 1,
                                 SQLITE_UTF8 | SQLITE_DETERMINISTIC, NULL,
                                 bf_crc32c_sse42, NULL, NULL);
    if (rc != SQLITE_OK) {
      return rc;
    }
  } else {
    int rc;
    rc = sqlite3_create_function(db, "unhex", 1,
                                 SQLITE_UTF8 | SQLITE_DETERMINISTIC, NULL,
                                 bf_unhex_sw, NULL, NULL);
    if (rc != SQLITE_OK) {
      return rc;
    }
    rc = sqlite3_create_function(db, "crc32c", 1,
                                 SQLITE_UTF8 | SQLITE_DETERMINISTIC, NULL,
                                 bf_crc32c_sw, NULL, NULL);
    if (rc != SQLITE_OK) {
      return rc;
    }
  }
#else
  {
    int rc;
    rc = sqlite3_create_function(db, "unhex", 1,
                                 SQLITE_UTF8 | SQLITE_DETERMINISTIC, NULL,
                                 bf_unhex_sw, NULL, NULL);
    if (rc != SQLITE_OK) {
      return rc;
    }
  }
#endif

  while (!RAND_status()) {
    unsigned char seed[32];
    sqlite3_randomness(sizeof seed, seed);
    RAND_seed(seed, sizeof seed);
  }

  for (int n = 0; func_table[n].name; n += 1) {
    int rc = sqlite3_create_function(
        db, func_table[n].name, func_table[n].nargs,
        SQLITE_UTF8 | SQLITE_DETERMINISTIC, NULL, func_table[n].fp, NULL, NULL);
    if (rc != SQLITE_OK) {
      return rc;
    }
  }
  return SQLITE_OK;
}



================================================
FILE: src/bloom_filter.c
================================================
/*
Copyright 2018 Shawn Wagner

Permission is hereby granted, free of charge, to any person obtaining
a copy of this software and associated documentation files (the
"Software"), to deal in the Software without restriction, including
without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject to
the following conditions:

The above copyright notice and this permission notice shall be
included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/

/* Bloom filter virtual table */

#include <math.h>
#include <stdlib.h>

/*
   SipHash reference C implementation

   Copyright (c) 2012-2016 Jean-Philippe Aumasson
   <jeanphilippe.aumasson@gmail.com>
   Copyright (c) 2012-2014 Daniel J. Bernstein <djb@cr.yp.to>

   To the extent possible under law, the author(s) have dedicated all copyright
   and related and neighboring rights to this software to the public domain
   worldwide. This software is distributed without any warranty.

   You should have received a copy of the CC0 Public Domain Dedication along
   with
   this software. If not, see
   <http://creativecommons.org/publicdomain/zero/1.0/>.
 */
#include <assert.h>
#include <stdint.h>
#include <stdio.h>
#include <string.h>

/* default: SipHash-2-4 */
#define cROUNDS 2
#define dROUNDS 4

#define ROTL(x, b) (uint64_t)(((x) << (b)) | ((x) >> (64 - (b))))

#define U32TO8_LE(p, v)                                                        \
  (p)[0] = (uint8_t)((v));                                                     \
  (p)[1] = (uint8_t)((v) >> 8);                                                \
  (p)[2] = (uint8_t)((v) >> 16);                                               \
  (p)[3] = (uint8_t)((v) >> 24);

#define U64TO8_LE(p, v)                                                        \
  U32TO8_LE((p), (uint32_t)((v)));                                             \
  U32TO8_LE((p) + 4, (uint32_t)((v) >> 32));

#define U8TO64_LE(p)                                                           \
  (((uint64_t)((p)[0])) | ((uint64_t)((p)[1]) << 8) |                          \
   ((uint64_t)((p)[2]) << 16) | ((uint64_t)((p)[3]) << 24) |                   \
   ((uint64_t)((p)[4]) << 32) | ((uint64_t)((p)[5]) << 40) |                   \
   ((uint64_t)((p)[6]) << 48) | ((uint64_t)((p)[7]) << 56))

#define SIPROUND                                                               \
  do {                                                                         \
    v0 += v1;                                                                  \
    v1 = ROTL(v1, 13);                                                         \
    v1 ^= v0;                                                                  \
    v0 = ROTL(v0, 32);                                                         \
    v2 += v3;                                                                  \
    v3 = ROTL(v3, 16);                                                         \
    v3 ^= v2;                                                                  \
    v0 += v3;                                                                  \
    v3 = ROTL(v3, 21);                                                         \
    v3 ^= v0;                                                                  \
    v2 += v1;                                                                  \
    v1 = ROTL(v1, 17);                                                         \
    v1 ^= v2;                                                                  \
    v2 = ROTL(v2, 32);                                                         \
  } while (0)

#ifdef DEBUG
#define TRACE                                                                  \
  do {                                                                         \
    printf("(%3d) v0 %08x %08x\n", (int)inlen, (uint32_t)(v0 >> 32),           \
           (uint32_t)v0);                                                      \
    printf("(%3d) v1 %08x %08x\n", (int)inlen, (uint32_t)(v1 >> 32),           \
           (uint32_t)v1);                                                      \
    printf("(%3d) v2 %08x %08x\n", (int)inlen, (uint32_t)(v2 >> 32),           \
           (uint32_t)v2);                                                      \
    printf("(%3d) v3 %08x %08x\n", (int)inlen, (uint32_t)(v3 >> 32),           \
           (uint32_t)v3);                                                      \
  } while (0)
#else
#define TRACE
#endif

static int siphash(const uint8_t *in, const size_t inlen, const uint8_t *k,
                   uint8_t *out, const size_t outlen) {

  assert((outlen == 8) || (outlen == 16));
  uint64_t v0 = 0x736f6d6570736575ULL;
  uint64_t v1 = 0x646f72616e646f6dULL;
  uint64_t v2 = 0x6c7967656e657261ULL;
  uint64_t v3 = 0x7465646279746573ULL;
  uint64_t k0 = U8TO64_LE(k);
  uint64_t k1 = U8TO64_LE(k + 8);
  uint64_t m;
  int i;
  const uint8_t *end = in + inlen - (inlen % sizeof(uint64_t));
  const int left = inlen & 7;
  uint64_t b = ((uint64_t)inlen) << 56;
  v3 ^= k1;
  v2 ^= k0;
  v1 ^= k1;
  v0 ^= k0;

  if (outlen == 16)
    v1 ^= 0xee;

  for (; in != end; in += 8) {
    m = U8TO64_LE(in);
    v3 ^= m;

    TRACE;
    for (i = 0; i < cROUNDS; ++i)
      SIPROUND;

    v0 ^= m;
  }

  switch (left) {
  case 7:
    b |= ((uint64_t)in[6]) << 48;
  case 6:
    b |= ((uint64_t)in[5]) << 40;
  case 5:
    b |= ((uint64_t)in[4]) << 32;
  case 4:
    b |= ((uint64_t)in[3]) << 24;
  case 3:
    b |= ((uint64_t)in[2]) << 16;
  case 2:
    b |= ((uint64_t)in[1]) << 8;
  case 1:
    b |= ((uint64_t)in[0]);
    break;
  case 0:
    break;
  }

  v3 ^= b;

  TRACE;
  for (i = 0; i < cROUNDS; ++i)
    SIPROUND;

  v0 ^= b;

  if (outlen == 16)
    v2 ^= 0xee;
  else
    v2 ^= 0xff;

  TRACE;
  for (i = 0; i < dROUNDS; ++i)
    SIPROUND;

  b = v0 ^ v1 ^ v2 ^ v3;
  U64TO8_LE(out, b);

  if (outlen == 8)
    return 0;

  v1 ^= 0xdd;

  TRACE;
  for (i = 0; i < dROUNDS; ++i)
    SIPROUND;

  b = v0 ^ v1 ^ v2 ^ v3;
  U64TO8_LE(out + 8, b);

  return 0;
}

#include <sqlite3ext.h>
SQLITE_EXTENSION_INIT1

static inline sqlite3_uint64 compute_bits(sqlite3_uint64 n, double p) {
  return ceil(-((n * log(p)) / (log(2.0) * log(2.0))));
}

static inline int compute_k(double p) { return round(-log2(p)); }

static inline int compute_bytes(sqlite3_uint64 bits) {
  int quo = bits / 8;
  int rem = bits % 8;
  if (rem) {
    quo += 1;
  }
  return quo;
}

static inline int bf_size(int nElements, double p) {
  return compute_bytes(compute_bits(nElements, p));
}

static int bf_create(sqlite3 *, void *, int, const char *const *,
                     sqlite3_vtab **, char **);
static int bf_connect(sqlite3 *, void *, int, const char *const *,
                      sqlite3_vtab **, char **);
static int bf_bestindex(sqlite3_vtab *, sqlite3_index_info *);
static int bf_disconnect(sqlite3_vtab *);
static int bf_destroy(sqlite3_vtab *);
static int bf_open(sqlite3_vtab *, sqlite3_vtab_cursor **);
static int bf_close(sqlite3_vtab_cursor *);
static int bf_filter(sqlite3_vtab_cursor *, int, const char *, int argc,
                     sqlite3_value **argv);
static int bf_next(sqlite3_vtab_cursor *);
static int bf_eof(sqlite3_vtab_cursor *);
static int bf_column(sqlite3_vtab_cursor *, sqlite3_context *, int);
static int bf_rowid(sqlite3_vtab_cursor *, sqlite3_int64 *);
static int bf_update(sqlite3_vtab *, int, sqlite3_value **, sqlite3_int64 *);
static int bf_rename(sqlite3_vtab *, const char *);

struct sqlite3_module bf_module = {
    1,        bf_create, bf_connect, bf_bestindex, bf_disconnect, bf_destroy,
    bf_open,  bf_close,  bf_filter,  bf_next,      bf_eof,        bf_column,
    bf_rowid, bf_update, NULL,       NULL,         NULL,          NULL,
    NULL,     bf_rename, NULL,       NULL,         NULL,
#if SQLITE_VERSION_NUMBER >= 3026000
    NULL, // xShadowName
#endif
};

struct bf_vtab {
  const sqlite3_module *pModule;
  int nRef;
  char *zErrMsg;
  char *zDBName;
  char *zName;
  char *zStorage;
  int nFilter;
  double p;
  int k;
  sqlite3 *db;
};

// CREATE VIRTUAL TABLE foo USING bloom_filter(nElemens, falseProb, kHashes)
static int bf_create(sqlite3 *db, void *pAux __attribute__((unused)), int argc,
                     char const *const *argv, sqlite3_vtab **ppVTab,
                     char **pzErr) {
  struct bf_vtab *vtab = sqlite3_malloc(sizeof *vtab);
  if (!vtab) {
    return SQLITE_NOMEM;
  }

  vtab->db = db;

  int nElems = 100;
  if (argc >= 4) {
    nElems = strtoull(argv[3], NULL, 0);
    if (nElems <= 0) {
      *pzErr = sqlite3_mprintf("Number of elements in filter must be positive");
      sqlite3_free(vtab);
      return SQLITE_ERROR;
    }
  }
  if (argc >= 5) {
    vtab->p = strtod(argv[4], NULL);
    if (vtab->p >= 1.0 || vtab->p <= 0.0) {
      *pzErr = sqlite3_mprintf("Probability must be in the range (0,1)");
      sqlite3_free(vtab);
      return SQLITE_ERROR;
    }
  } else {
    vtab->p = 0.01;
  }
  vtab->nFilter = bf_size(nElems, vtab->p);
  if (argc == 6) {
    vtab->k = strtol(argv[5], NULL, 0);
    if (vtab->k <= 0) {
      *pzErr = sqlite3_mprintf("Number of hash functions must be positive.");
      sqlite3_free(vtab);
      return SQLITE_ERROR;
    }
  } else {
    vtab->k = compute_k(vtab->p);
  }

  char *storage =
      sqlite3_mprintf("CREATE TABLE \"%s\".\"%s_storage\"(data "
                      "BLOB, p REAL, n INTEGER, m INTEGER, k INTEGER)",
                      argv[1], argv[2]);
  int rc = sqlite3_exec(db, storage, NULL, NULL, pzErr);
  sqlite3_free(storage);
  if (rc != SQLITE_OK) {
    sqlite3_free(vtab);
    return rc;
  }

  storage =
      sqlite3_mprintf("INSERT INTO \"%s\".\"%s_storage\"(rowid, data, p, n, m, "
                      "k) VALUES (1, ?, %f, %d, %llu, %d)",
                      argv[1], argv[2], vtab->p, nElems,
                      (sqlite3_uint64)vtab->nFilter * 8, vtab->k);

  sqlite3_stmt *inserter;
  rc = sqlite3_prepare_v2(db, storage, -1, &inserter, NULL);
  sqlite3_free(storage);
  if (rc != SQLITE_OK) {
    sqlite3_free(vtab);
    return rc;
  }

  sqlite3_bind_zeroblob(inserter, 1, vtab->nFilter);
  rc = sqlite3_step(inserter);
  sqlite3_finalize(inserter);
  if (rc != SQLITE_DONE) {
    sqlite3_free(vtab);
    return rc;
  }

  rc = sqlite3_declare_vtab(db, "CREATE TABLE x(present, word HIDDEN "
                                "NOT NULL PRIMARY KEY) WITHOUT ROWID");
  if (rc != SQLITE_OK) {
    storage =
        sqlite3_mprintf("DROP TABLE \"%s\".\"%s_storage\"", argv[1], argv[2]);
    sqlite3_exec(db, storage, NULL, NULL, NULL);
    sqlite3_free(storage);
    sqlite3_free(vtab);
    return rc;
  }

  vtab->zDBName = sqlite3_mprintf("%s", argv[1]);
  if (!vtab->zDBName) {
    sqlite3_free(vtab);
    return SQLITE_NOMEM;
  }
  vtab->zName = sqlite3_mprintf("%s", argv[2]);
  if (!vtab->zName) {
    sqlite3_free(vtab->zDBName);
    sqlite3_free(vtab);
    return SQLITE_NOMEM;
  }
  vtab->zStorage = sqlite3_mprintf("%s_storage", argv[2]);
  if (!vtab->zName) {
    sqlite3_free(vtab->zName);
    sqlite3_free(vtab->zDBName);
    sqlite3_free(vtab);
    return SQLITE_NOMEM;
  }

  *ppVTab = (struct sqlite3_vtab *)vtab;
  return SQLITE_OK;
}

static void free_bf_vtab(struct bf_vtab *vtab) {
  sqlite3_free(vtab->zErrMsg);
  sqlite3_free(vtab->zDBName);
  sqlite3_free(vtab->zName);
  sqlite3_free(vtab->zStorage);
  sqlite3_free(vtab);
}

static int bf_connect(sqlite3 *db, void *pAux __attribute__((unused)),
                      int argc __attribute__((unused)), const char *const *argv,
                      sqlite3_vtab **ppVTab,
                      char **pzErr __attribute__((unused))) {
  struct bf_vtab *vtab = sqlite3_malloc(sizeof *vtab);
  if (!vtab) {
    return SQLITE_NOMEM;
  }
  vtab->db = db;

  int rc = sqlite3_declare_vtab(db, "CREATE TABLE x(present, word HIDDEN  "
                                    "NOT NULL PRIMARY KEY) WITHOUT ROWID");
  if (rc != SQLITE_OK) {
    sqlite3_free(vtab);
    return rc;
  }

  char *load_query = sqlite3_mprintf(
      "SELECT m/8, p, k FROM \"%s\".\"%s_storage\" WHERE rowid = 1", argv[1],
      argv[2]);
  if (!load_query) {
    sqlite3_free(vtab);
    return SQLITE_NOMEM;
  }

  sqlite3_stmt *loader;
  rc = sqlite3_prepare_v2(db, load_query, -1, &loader, NULL);
  sqlite3_free(load_query);
  if (rc != SQLITE_OK) {
    sqlite3_free(vtab);
    return rc;
  }

  rc = sqlite3_step(loader);
  if (rc != SQLITE_ROW) {
    sqlite3_finalize(loader);
    sqlite3_free(vtab);
    if (rc == SQLITE_DONE) {
      rc = SQLITE_CORRUPT_VTAB;
    }
    return rc;
  }

  vtab->nFilter = sqlite3_column_int(loader, 0);
  vtab->p = sqlite3_column_double(loader, 1);
  vtab->k = sqlite3_column_int(loader, 2);
  sqlite3_finalize(loader);

  vtab->zDBName = sqlite3_mprintf("%s", argv[1]);
  if (!vtab->zDBName) {
    sqlite3_free(vtab);
    return SQLITE_NOMEM;
  }
  vtab->zName = sqlite3_mprintf("%s", argv[2]);
  if (!vtab->zName) {
    sqlite3_free(vtab->zDBName);
    sqlite3_free(vtab);
    return SQLITE_NOMEM;
  }
  vtab->zStorage = sqlite3_mprintf("%s_storage", argv[2]);
  if (!vtab->zStorage) {
    sqlite3_free(vtab->zName);
    sqlite3_free(vtab->zDBName);
    sqlite3_free(vtab);
    return SQLITE_NOMEM;
  }

  *ppVTab = (struct sqlite3_vtab *)vtab;
  return SQLITE_OK;
}

static int bf_bestindex(struct sqlite3_vtab *pVTab, sqlite3_index_info *idx) {
  struct bf_vtab *vtab = (struct bf_vtab *)pVTab;

  for (int n = 0; n < idx->nConstraint; n += 1) {
    if (!idx->aConstraint[n].usable) {
      continue;
    }
    if (idx->aConstraint[n].iColumn == 1 &&
        idx->aConstraint[n].op == SQLITE_INDEX_CONSTRAINT_EQ) {
      idx->aConstraintUsage[n].argvIndex = 1;
    }
  }
  idx->orderByConsumed = 1;
  if (sqlite3_libversion_number() >= 3008002) {
    idx->estimatedRows = 1;
  }
  if (sqlite3_libversion_number() >= 3009000) {
    idx->idxFlags = SQLITE_INDEX_SCAN_UNIQUE;
  }
  idx->estimatedCost = vtab->k;
  return SQLITE_OK;
}

static int bf_disconnect(struct sqlite3_vtab *pVTab) {
  free_bf_vtab((struct bf_vtab *)pVTab);
  return SQLITE_OK;
}

static int bf_destroy(struct sqlite3_vtab *pVTab) {
  struct bf_vtab *vtab = (struct bf_vtab *)pVTab;

  char *deleter = sqlite3_mprintf("DROP TABLE \"%s\".\"%s\"", vtab->zDBName,
                                  vtab->zStorage);
  int rc = sqlite3_exec(vtab->db, deleter, NULL, NULL, NULL);
  sqlite3_free(deleter);
  if (rc != SQLITE_OK) {
    return rc;
  }

  free_bf_vtab(vtab);
  return SQLITE_OK;
}

struct bf_cursor {
  struct bf_vtab *vtab;
  sqlite3_value *orig;
  _Bool found;
};

static int bf_open(sqlite3_vtab *pVTab __attribute__((unused)),
                   sqlite3_vtab_cursor **ppCursor) {
  struct bf_cursor *c = sqlite3_malloc(sizeof *c);
  if (!c) {
    return SQLITE_NOMEM;
  }
  c->orig = NULL;
  c->found = 0;
  *ppCursor = (struct sqlite3_vtab_cursor *)c;
  return SQLITE_OK;
}

static int bf_close(sqlite3_vtab_cursor *pCursor) {
  struct bf_cursor *c = (struct bf_cursor *)pCursor;
  sqlite3_value_free(c->orig);
  sqlite3_free(c);
  return SQLITE_OK;
}

static int bf_eof(sqlite3_vtab_cursor *pCursor) {
  struct bf_cursor *c = (struct bf_cursor *)pCursor;
  return c->found == 0;
}

static inline uint64_t calc_hash(const uint8_t *data, size_t len, int k) {
  uint64_t seed[2] = {~((uint64_t)k), k};
  uint64_t hash;
  siphash(data, len, (const uint8_t *)seed, (uint8_t *)&hash, sizeof hash);
  return hash;
}

static int bf_filter(sqlite3_vtab_cursor *pCursor,
                     int idxNum __attribute__((unused)),
                     const char *idxStr __attribute__((unused)), int argc,
                     sqlite3_value **argv) {
  struct bf_cursor *c = (struct bf_cursor *)pCursor;
  int rc;

  if (argc != 1) {
    return SQLITE_OK;
  }

  c->orig = sqlite3_value_dup(argv[0]);
  const unsigned char *blob = sqlite3_value_blob(argv[0]);
  int len = sqlite3_value_bytes(argv[0]);

  do {
    sqlite3_blob *b;
    rc = sqlite3_blob_open(c->vtab->db, c->vtab->zDBName, c->vtab->zStorage,
                           "data", 1, 0, &b);
    if (rc != SQLITE_OK) {
      return rc;
    }

    for (int n = 0; n < c->vtab->k; n += 1) {
      uint64_t hash = calc_hash(blob, len, n);
      hash %= (uint64_t)c->vtab->nFilter * 8;
      int bytepos = hash / 8;
      int bitpos = hash % 8;
      unsigned char byte;
      rc = sqlite3_blob_read(b, &byte, 1, bytepos);
      if (rc == SQLITE_OK) {
        c->found = byte & (1 << bitpos);
        if (!c->found) {
          sqlite3_blob_close(b);
          return SQLITE_OK;
        }
      } else if (rc == SQLITE_ABORT) {
        break;
      } else {
        sqlite3_blob_close(b);
        return rc;
      }
    }
    sqlite3_blob_close(b);
  } while (rc == SQLITE_ABORT);
  return rc;
}

static int bf_next(sqlite3_vtab_cursor *pCursor) {
  struct bf_cursor *c = (struct bf_cursor *)pCursor;
  c->found = 0;
  return SQLITE_OK;
}

static int bf_column(sqlite3_vtab_cursor *pCursor, sqlite3_context *ctx,
                     int n) {
  struct bf_cursor *c = (struct bf_cursor *)pCursor;
  assert(n == 0 || n == 1);

  if (n == 0) {
    sqlite3_result_int(ctx, c->found);
  } else if (n == 1) {
    sqlite3_result_value(ctx, c->orig);
  }
  return SQLITE_OK;
}

static int bf_rowid(sqlite3_vtab_cursor *pCursor __attribute__((unused)),
                    sqlite_int64 *pRowid __attribute__((unused))) {
  return SQLITE_OK;
}

static int bf_update(sqlite3_vtab *pVTab, int argc, sqlite3_value **argv,
                     sqlite_int64 *pRowid __attribute__((unused))) {
  struct bf_vtab *vtab = (struct bf_vtab *)pVTab;

  if (sqlite3_value_type(argv[0]) != SQLITE_NULL) {
    if (vtab->zErrMsg) {
      sqlite3_free(vtab->zErrMsg);
    }
    if (argc == 1) {
      vtab->zErrMsg =
          sqlite3_mprintf("bloom_filter elements cannot be deleted.");
    } else {
      vtab->zErrMsg =
          sqlite3_mprintf("bloom_filter elements cannot be updated.");
    }
    return SQLITE_ERROR;
  }

  assert(argc == 4);

  const uint8_t *blob = sqlite3_value_blob(argv[2]);
  size_t len = sqlite3_value_bytes(argv[2]);

  int rc;
  do {
    sqlite3_blob *b;
    rc = sqlite3_blob_open(vtab->db, vtab->zDBName, vtab->zStorage, "data", 1,
                           1, &b);
    if (rc != SQLITE_OK) {
      return rc;
    }
    for (int n = 0; n < vtab->k; n += 1) {
      uint64_t hash = calc_hash(blob, len, n);
      hash %= (uint64_t)vtab->nFilter * 8;
      int bytepos = hash / 8;
      int bitpos = hash % 8;
      unsigned char byte;
      rc = sqlite3_blob_read(b, &byte, 1, bytepos);
      if (rc == SQLITE_OK) {
        byte |= 1 << bitpos;
        rc = sqlite3_blob_write(b, &byte, 1, bytepos);
        if (rc == SQLITE_ABORT) {
          break;
        } else if (rc != SQLITE_OK) {
          sqlite3_blob_close(b);
          return rc;
        }
      } else if (rc == SQLITE_ABORT) {
        break;
      } else {
        sqlite3_blob_close(b);
        return rc;
      }
    }
    sqlite3_blob_close(b);
  } while (rc == SQLITE_ABORT);

  return rc;
}

static int bf_rename(sqlite3_vtab *pVTab, const char *zNew) {
  struct bf_vtab *vtab = (struct bf_vtab *)pVTab;

  char *renamer =
      sqlite3_mprintf("ALTER TABLE \"%s\".\"%s\" RENAME TO \"%s_storage\"",
                      vtab->zDBName, vtab->zStorage, zNew);
  if (!renamer) {
    return SQLITE_NOMEM;
  }
  if (vtab->zErrMsg) {
    sqlite3_free(vtab->zErrMsg);
    vtab->zErrMsg = NULL;
  }
  int rc = sqlite3_exec(vtab->db, renamer, NULL, NULL, &(vtab->zErrMsg));
  sqlite3_free(renamer);
  if (rc == SQLITE_OK) {
    sqlite3_free(vtab->zName);
    vtab->zName = sqlite3_mprintf("%s", zNew);
    if (!vtab->zName) {
      return SQLITE_NOMEM;
    }
    sqlite3_free(vtab->zStorage);
    vtab->zStorage = sqlite3_mprintf("%s_storage", zNew);
    if (!vtab->zStorage) {
      return SQLITE_NOMEM;
    }
  }
  return rc;
}

#ifdef _WIN32
_declspec(dllexport)
#endif
    int sqlite3_bloomfilter_init(sqlite3 *db,
                                 char **pzErrMsg __attribute__((unused)),
                                 const sqlite3_api_routines *pApi) {
  SQLITE_EXTENSION_INIT2(pApi);
  return sqlite3_create_module(db, "bloom_filter1", &bf_module, NULL);
}



================================================
FILE: src/cJSON.c
================================================
/*
  Copyright (c) 2009-2017 Dave Gamble and cJSON contributors

  Permission is hereby granted, free of charge, to any person obtaining a copy
  of this software and associated documentation files (the "Software"), to deal
  in the Software without restriction, including without limitation the rights
  to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
  copies of the Software, and to permit persons to whom the Software is
  furnished to do so, subject to the following conditions:

  The above copyright notice and this permission notice shall be included in
  all copies or substantial portions of the Software.

  THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
  IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
  FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
  AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
  LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
  OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
  THE SOFTWARE.
*/

/* cJSON */
/* JSON parser in C. */

/* disable warnings about old C89 functions in MSVC */
#if !defined(_CRT_SECURE_NO_DEPRECATE) && defined(_MSC_VER)
#define _CRT_SECURE_NO_DEPRECATE
#endif

#ifdef __GNUC__
#pragma GCC visibility push(default)
#endif
#if defined(_MSC_VER)
#pragma warning (push)
/* disable warning about single line comments in system headers */
#pragma warning (disable : 4001)
#endif

#include <string.h>
#include <stdio.h>
#include <math.h>
#include <stdlib.h>
#include <limits.h>
#include <ctype.h>

#ifdef ENABLE_LOCALES
#include <locale.h>
#endif

#if defined(_MSC_VER)
#pragma warning (pop)
#endif
#ifdef __GNUC__
#pragma GCC visibility pop
#endif

#include "cJSON.h"

#ifndef __bool_true_false_are_defined
/* define our own boolean type */
#define true ((cJSON_bool)1)
#define false ((cJSON_bool)0)
#endif

typedef struct {
    const unsigned char *json;
    size_t position;
} error;
static error global_error = { NULL, 0 };

CJSON_PUBLIC(const char *) cJSON_GetErrorPtr(void)
{
    return (const char*) (global_error.json + global_error.position);
}

CJSON_PUBLIC(char *) cJSON_GetStringValue(cJSON *item) {
    if (!cJSON_IsString(item)) {
        return NULL;
    }

    return item->valuestring;
}

/* This is a safeguard to prevent copy-pasters from using incompatible C and header files */
#if (CJSON_VERSION_MAJOR != 1) || (CJSON_VERSION_MINOR != 7) || (CJSON_VERSION_PATCH != 7)
    #error cJSON.h and cJSON.c have different versions. Make sure that both have the same.
#endif

CJSON_PUBLIC(const char*) cJSON_Version(void)
{
    static char version[15];
    sprintf(version, "%i.%i.%i", CJSON_VERSION_MAJOR, CJSON_VERSION_MINOR, CJSON_VERSION_PATCH);

    return version;
}

/* Case insensitive string comparison, doesn't consider two NULL pointers equal though */
static int case_insensitive_strcmp(const unsigned char *string1, const unsigned char *string2)
{
    if ((string1 == NULL) || (string2 == NULL))
    {
        return 1;
    }

    if (string1 == string2)
    {
        return 0;
    }

    for(; tolower(*string1) == tolower(*string2); (void)string1++, string2++)
    {
        if (*string1 == '\0')
        {
            return 0;
        }
    }

    return tolower(*string1) - tolower(*string2);
}

typedef struct internal_hooks
{
    void *(*allocate)(size_t size);
    void (*deallocate)(void *pointer);
    void *(*reallocate)(void *pointer, size_t size);
} internal_hooks;

#if defined(_MSC_VER)
/* work around MSVC error C2322: '...' address of dillimport '...' is not static */
static void *internal_malloc(size_t size)
{
    return malloc(size);
}
static void internal_free(void *pointer)
{
    free(pointer);
}
static void *internal_realloc(void *pointer, size_t size)
{
    return realloc(pointer, size);
}
#else
#define internal_malloc malloc
#define internal_free free
#define internal_realloc realloc
#endif

static internal_hooks global_hooks = { internal_malloc, internal_free, internal_realloc };

static unsigned char* cJSON_strdup(const unsigned char* string, const internal_hooks * const hooks)
{
    size_t length = 0;
    unsigned char *copy = NULL;

    if (string == NULL)
    {
        return NULL;
    }

    length = strlen((const char*)string) + sizeof("");
    copy = (unsigned char*)hooks->allocate(length);
    if (copy == NULL)
    {
        return NULL;
    }
    memcpy(copy, string, length);

    return copy;
}

CJSON_PUBLIC(void) cJSON_InitHooks(cJSON_Hooks* hooks)
{
    if (hooks == NULL)
    {
        /* Reset hooks */
        global_hooks.allocate = malloc;
        global_hooks.deallocate = free;
        global_hooks.reallocate = realloc;
        return;
    }

    global_hooks.allocate = malloc;
    if (hooks->malloc_fn != NULL)
    {
        global_hooks.allocate = hooks->malloc_fn;
    }

    global_hooks.deallocate = free;
    if (hooks->free_fn != NULL)
    {
        global_hooks.deallocate = hooks->free_fn;
    }

    /* use realloc only if both free and malloc are used */
    global_hooks.reallocate = NULL;
    if ((global_hooks.allocate == malloc) && (global_hooks.deallocate == free))
    {
        global_hooks.reallocate = realloc;
    }
}

/* Internal constructor. */
static cJSON *cJSON_New_Item(const internal_hooks * const hooks)
{
    cJSON* node = (cJSON*)hooks->allocate(sizeof(cJSON));
    if (node)
    {
        memset(node, '\0', sizeof(cJSON));
    }

    return node;
}

/* Delete a cJSON structure. */
CJSON_PUBLIC(void) cJSON_Delete(cJSON *item)
{
    cJSON *next = NULL;
    while (item != NULL)
    {
        next = item->next;
        if (!(item->type & cJSON_IsReference) && (item->child != NULL))
        {
            cJSON_Delete(item->child);
        }
        if (!(item->type & cJSON_IsReference) && (item->valuestring != NULL))
        {
            global_hooks.deallocate(item->valuestring);
        }
        if (!(item->type & cJSON_StringIsConst) && (item->string != NULL))
        {
            global_hooks.deallocate(item->string);
        }
        global_hooks.deallocate(item);
        item = next;
    }
}

/* get the decimal point character of the current locale */
static unsigned char get_decimal_point(void)
{
#ifdef ENABLE_LOCALES
    struct lconv *lconv = localeconv();
    return (unsigned char) lconv->decimal_point[0];
#else
    return '.';
#endif
}

typedef struct
{
    const unsigned char *content;
    size_t length;
    size_t offset;
    size_t depth; /* How deeply nested (in arrays/objects) is the input at the current offset. */
    internal_hooks hooks;
} parse_buffer;

/* check if the given size is left to read in a given parse buffer (starting with 1) */
#define can_read(buffer, size) ((buffer != NULL) && (((buffer)->offset + size) <= (buffer)->length))
/* check if the buffer can be accessed at the given index (starting with 0) */
#define can_access_at_index(buffer, index) ((buffer != NULL) && (((buffer)->offset + index) < (buffer)->length))
#define cannot_access_at_index(buffer, index) (!can_access_at_index(buffer, index))
/* get a pointer to the buffer at the position */
#define buffer_at_offset(buffer) ((buffer)->content + (buffer)->offset)

/* Parse the input text to generate a number, and populate the result into item. */
static cJSON_bool parse_number(cJSON * const item, parse_buffer * const input_buffer)
{
    double number = 0;
    unsigned char *after_end = NULL;
    unsigned char number_c_string[64];
    unsigned char decimal_point = get_decimal_point();
    size_t i = 0;

    if ((input_buffer == NULL) || (input_buffer->content == NULL))
    {
        return false;
    }

    /* copy the number into a temporary buffer and replace '.' with the decimal point
     * of the current locale (for strtod)
     * This also takes care of '\0' not necessarily being available for marking the end of the input */
    for (i = 0; (i < (sizeof(number_c_string) - 1)) && can_access_at_index(input_buffer, i); i++)
    {
        switch (buffer_at_offset(input_buffer)[i])
        {
            case '0':
            case '1':
            case '2':
            case '3':
            case '4':
            case '5':
            case '6':
            case '7':
            case '8':
            case '9':
            case '+':
            case '-':
            case 'e':
            case 'E':
                number_c_string[i] = buffer_at_offset(input_buffer)[i];
                break;

            case '.':
                number_c_string[i] = decimal_point;
                break;

            default:
                goto loop_end;
        }
    }
loop_end:
    number_c_string[i] = '\0';

    number = strtod((const char*)number_c_string, (char**)&after_end);
    if (number_c_string == after_end)
    {
        return false; /* parse_error */
    }

    item->valuedouble = number;

    /* use saturation in case of overflow */
    if (number >= INT_MAX)
    {
        item->valueint = INT_MAX;
    }
    else if (number <= INT_MIN)
    {
        item->valueint = INT_MIN;
    }
    else
    {
        item->valueint = (int)number;
    }

    item->type = cJSON_Number;

    input_buffer->offset += (size_t)(after_end - number_c_string);
    return true;
}

/* don't ask me, but the original cJSON_SetNumberValue returns an integer or double */
CJSON_PUBLIC(double) cJSON_SetNumberHelper(cJSON *object, double number)
{
    if (number >= INT_MAX)
    {
        object->valueint = INT_MAX;
    }
    else if (number <= INT_MIN)
    {
        object->valueint = INT_MIN;
    }
    else
    {
        object->valueint = (int)number;
    }

    return object->valuedouble = number;
}

typedef struct
{
    unsigned char *buffer;
    size_t length;
    size_t offset;
    size_t depth; /* current nesting depth (for formatted printing) */
    cJSON_bool noalloc;
    cJSON_bool format; /* is this print a formatted print */
    internal_hooks hooks;
} printbuffer;

/* realloc printbuffer if necessary to have at least "needed" bytes more */
static unsigned char* ensure(printbuffer * const p, size_t needed)
{
    unsigned char *newbuffer = NULL;
    size_t newsize = 0;

    if ((p == NULL) || (p->buffer == NULL))
    {
        return NULL;
    }

    if ((p->length > 0) && (p->offset >= p->length))
    {
        /* make sure that offset is valid */
        return NULL;
    }

    if (needed > INT_MAX)
    {
        /* sizes bigger than INT_MAX are currently not supported */
        return NULL;
    }

    needed += p->offset + 1;
    if (needed <= p->length)
    {
        return p->buffer + p->offset;
    }

    if (p->noalloc) {
        return NULL;
    }

    /* calculate new buffer size */
    if (needed > (INT_MAX / 2))
    {
        /* overflow of int, use INT_MAX if possible */
        if (needed <= INT_MAX)
        {
            newsize = INT_MAX;
        }
        else
        {
            return NULL;
        }
    }
    else
    {
        newsize = needed * 2;
    }

    if (p->hooks.reallocate != NULL)
    {
        /* reallocate with realloc if available */
        newbuffer = (unsigned char*)p->hooks.reallocate(p->buffer, newsize);
        if (newbuffer == NULL)
        {
            p->hooks.deallocate(p->buffer);
            p->length = 0;
            p->buffer = NULL;

            return NULL;
        }
    }
    else
    {
        /* otherwise reallocate manually */
        newbuffer = (unsigned char*)p->hooks.allocate(newsize);
        if (!newbuffer)
        {
            p->hooks.deallocate(p->buffer);
            p->length = 0;
            p->buffer = NULL;

            return NULL;
        }
        if (newbuffer)
        {
            memcpy(newbuffer, p->buffer, p->offset + 1);
        }
        p->hooks.deallocate(p->buffer);
    }
    p->length = newsize;
    p->buffer = newbuffer;

    return newbuffer + p->offset;
}

/* calculate the new length of the string in a printbuffer and update the offset */
static void update_offset(printbuffer * const buffer)
{
    const unsigned char *buffer_pointer = NULL;
    if ((buffer == NULL) || (buffer->buffer == NULL))
    {
        return;
    }
    buffer_pointer = buffer->buffer + buffer->offset;

    buffer->offset += strlen((const char*)buffer_pointer);
}

/* Render the number nicely from the given item into a string. */
static cJSON_bool print_number(const cJSON * const item, printbuffer * const output_buffer)
{
    unsigned char *output_pointer = NULL;
    double d = item->valuedouble;
    int length = 0;
    size_t i = 0;
    unsigned char number_buffer[26]; /* temporary buffer to print the number into */
    unsigned char decimal_point = get_decimal_point();
    double test;

    if (output_buffer == NULL)
    {
        return false;
    }

    /* This checks for NaN and Infinity */
    if ((d * 0) != 0)
    {
        length = sprintf((char*)number_buffer, "null");
    }
    else
    {
        /* Try 15 decimal places of precision to avoid nonsignificant nonzero digits */
        length = sprintf((char*)number_buffer, "%1.15g", d);

        /* Check whether the original double can be recovered */
        if ((sscanf((char*)number_buffer, "%lg", &test) != 1) || ((double)test != d))
        {
            /* If not, print with 17 decimal places of precision */
            length = sprintf((char*)number_buffer, "%1.17g", d);
        }
    }

    /* sprintf failed or buffer overrun occured */
    if ((length < 0) || (length > (int)(sizeof(number_buffer) - 1)))
    {
        return false;
    }

    /* reserve appropriate space in the output */
    output_pointer = ensure(output_buffer, (size_t)length + sizeof(""));
    if (output_pointer == NULL)
    {
        return false;
    }

    /* copy the printed number to the output and replace locale
     * dependent decimal point with '.' */
    for (i = 0; i < ((size_t)length); i++)
    {
        if (number_buffer[i] == decimal_point)
        {
            output_pointer[i] = '.';
            continue;
        }

        output_pointer[i] = number_buffer[i];
    }
    output_pointer[i] = '\0';

    output_buffer->offset += (size_t)length;

    return true;
}

/* parse 4 digit hexadecimal number */
static unsigned parse_hex4(const unsigned char * const input)
{
    unsigned int h = 0;
    size_t i = 0;

    for (i = 0; i < 4; i++)
    {
        /* parse digit */
        if ((input[i] >= '0') && (input[i] <= '9'))
        {
            h += (unsigned int) input[i] - '0';
        }
        else if ((input[i] >= 'A') && (input[i] <= 'F'))
        {
            h += (unsigned int) 10 + input[i] - 'A';
        }
        else if ((input[i] >= 'a') && (input[i] <= 'f'))
        {
            h += (unsigned int) 10 + input[i] - 'a';
        }
        else /* invalid */
        {
            return 0;
        }

        if (i < 3)
        {
            /* shift left to make place for the next nibble */
            h = h << 4;
        }
    }

    return h;
}

/* converts a UTF-16 literal to UTF-8
 * A literal can be one or two sequences of the form \uXXXX */
static unsigned char utf16_literal_to_utf8(const unsigned char * const input_pointer, const unsigned char * const input_end, unsigned char **output_pointer)
{
    long unsigned int codepoint = 0;
    unsigned int first_code = 0;
    const unsigned char *first_sequence = input_pointer;
    unsigned char utf8_length = 0;
    unsigned char utf8_position = 0;
    unsigned char sequence_length = 0;
    unsigned char first_byte_mark = 0;

    if ((input_end - first_sequence) < 6)
    {
        /* input ends unexpectedly */
        goto fail;
    }

    /* get the first utf16 sequence */
    first_code = parse_hex4(first_sequence + 2);

    /* check that the code is valid */
    if (((first_code >= 0xDC00) && (first_code <= 0xDFFF)))
    {
        goto fail;
    }

    /* UTF16 surrogate pair */
    if ((first_code >= 0xD800) && (first_code <= 0xDBFF))
    {
        const unsigned char *second_sequence = first_sequence + 6;
        unsigned int second_code = 0;
        sequence_length = 12; /* \uXXXX\uXXXX */

        if ((input_end - second_sequence) < 6)
        {
            /* input ends unexpectedly */
            goto fail;
        }

        if ((second_sequence[0] != '\\') || (second_sequence[1] != 'u'))
        {
            /* missing second half of the surrogate pair */
            goto fail;
        }

        /* get the second utf16 sequence */
        second_code = parse_hex4(second_sequence + 2);
        /* check that the code is valid */
        if ((second_code < 0xDC00) || (second_code > 0xDFFF))
        {
            /* invalid second half of the surrogate pair */
            goto fail;
        }


        /* calculate the unicode codepoint from the surrogate pair */
        codepoint = 0x10000 + (((first_code & 0x3FF) << 10) | (second_code & 0x3FF));
    }
    else
    {
        sequence_length = 6; /* \uXXXX */
        codepoint = first_code;
    }

    /* encode as UTF-8
     * takes at maximum 4 bytes to encode:
     * 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx */
    if (codepoint < 0x80)
    {
        /* normal ascii, encoding 0xxxxxxx */
        utf8_length = 1;
    }
    else if (codepoint < 0x800)
    {
        /* two bytes, encoding 110xxxxx 10xxxxxx */
        utf8_length = 2;
        first_byte_mark = 0xC0; /* 11000000 */
    }
    else if (codepoint < 0x10000)
    {
        /* three bytes, encoding 1110xxxx 10xxxxxx 10xxxxxx */
        utf8_length = 3;
        first_byte_mark = 0xE0; /* 11100000 */
    }
    else if (codepoint <= 0x10FFFF)
    {
        /* four bytes, encoding 1110xxxx 10xxxxxx 10xxxxxx 10xxxxxx */
        utf8_length = 4;
        first_byte_mark = 0xF0; /* 11110000 */
    }
    else
    {
        /* invalid unicode codepoint */
        goto fail;
    }

    /* encode as utf8 */
    for (utf8_position = (unsigned char)(utf8_length - 1); utf8_position > 0; utf8_position--)
    {
        /* 10xxxxxx */
        (*output_pointer)[utf8_position] = (unsigned char)((codepoint | 0x80) & 0xBF);
        codepoint >>= 6;
    }
    /* encode first byte */
    if (utf8_length > 1)
    {
        (*output_pointer)[0] = (unsigned char)((codepoint | first_byte_mark) & 0xFF);
    }
    else
    {
        (*output_pointer)[0] = (unsigned char)(codepoint & 0x7F);
    }

    *output_pointer += utf8_length;

    return sequence_length;

fail:
    return 0;
}

/* Parse the input text into an unescaped cinput, and populate item. */
static cJSON_bool parse_string(cJSON * const item, parse_buffer * const input_buffer)
{
    const unsigned char *input_pointer = buffer_at_offset(input_buffer) + 1;
    const unsigned char *input_end = buffer_at_offset(input_buffer) + 1;
    unsigned char *output_pointer = NULL;
    unsigned char *output = NULL;

    /* not a string */
    if (buffer_at_offset(input_buffer)[0] != '\"')
    {
        goto fail;
    }

    {
        /* calculate approximate size of the output (overestimate) */
        size_t allocation_length = 0;
        size_t skipped_bytes = 0;
        while (((size_t)(input_end - input_buffer->content) < input_buffer->length) && (*input_end != '\"'))
        {
            /* is escape sequence */
            if (input_end[0] == '\\')
            {
                if ((size_t)(input_end + 1 - input_buffer->content) >= input_buffer->length)
                {
                    /* prevent buffer overflow when last input character is a backslash */
                    goto fail;
                }
                skipped_bytes++;
                input_end++;
            }
            input_end++;
        }
        if (((size_t)(input_end - input_buffer->content) >= input_buffer->length) || (*input_end != '\"'))
        {
            goto fail; /* string ended unexpectedly */
        }

        /* This is at most how much we need for the output */
        allocation_length = (size_t) (input_end - buffer_at_offset(input_buffer)) - skipped_bytes;
        output = (unsigned char*)input_buffer->hooks.allocate(allocation_length + sizeof(""));
        if (output == NULL)
        {
            goto fail; /* allocation failure */
        }
    }

    output_pointer = output;
    /* loop through the string literal */
    while (input_pointer < input_end)
    {
        if (*input_pointer != '\\')
        {
            *output_pointer++ = *input_pointer++;
        }
        /* escape sequence */
        else
        {
            unsigned char sequence_length = 2;
            if ((input_end - input_pointer) < 1)
            {
                goto fail;
            }

            switch (input_pointer[1])
            {
                case 'b':
                    *output_pointer++ = '\b';
                    break;
                case 'f':
                    *output_pointer++ = '\f';
                    break;
                case 'n':
                    *output_pointer++ = '\n';
                    break;
                case 'r':
                    *output_pointer++ = '\r';
                    break;
                case 't':
                    *output_pointer++ = '\t';
                    break;
                case '\"':
                case '\\':
                case '/':
                    *output_pointer++ = input_pointer[1];
                    break;

                /* UTF-16 literal */
                case 'u':
                    sequence_length = utf16_literal_to_utf8(input_pointer, input_end, &output_pointer);
                    if (sequence_length == 0)
                    {
                        /* failed to convert UTF16-literal to UTF-8 */
                        goto fail;
                    }
                    break;

                default:
                    goto fail;
            }
            input_pointer += sequence_length;
        }
    }

    /* zero terminate the output */
    *output_pointer = '\0';

    item->type = cJSON_String;
    item->valuestring = (char*)output;

    input_buffer->offset = (size_t) (input_end - input_buffer->content);
    input_buffer->offset++;

    return true;

fail:
    if (output != NULL)
    {
        input_buffer->hooks.deallocate(output);
    }

    if (input_pointer != NULL)
    {
        input_buffer->offset = (size_t)(input_pointer - input_buffer->content);
    }

    return false;
}

/* Render the cstring provided to an escaped version that can be printed. */
static cJSON_bool print_string_ptr(const unsigned char * const input, printbuffer * const output_buffer)
{
    const unsigned char *input_pointer = NULL;
    unsigned char *output = NULL;
    unsigned char *output_pointer = NULL;
    size_t output_length = 0;
    /* numbers of additional characters needed for escaping */
    size_t escape_characters = 0;

    if (output_buffer == NULL)
    {
        return false;
    }

    /* empty string */
    if (input == NULL)
    {
        output = ensure(output_buffer, sizeof("\"\""));
        if (output == NULL)
        {
            return false;
        }
        strcpy((char*)output, "\"\"");

        return true;
    }

    /* set "flag" to 1 if something needs to be escaped */
    for (input_pointer = input; *input_pointer; input_pointer++)
    {
        switch (*input_pointer)
        {
            case '\"':
            case '\\':
            case '\b':
            case '\f':
            case '\n':
            case '\r':
            case '\t':
                /* one character escape sequence */
                escape_characters++;
                break;
            default:
                if (*input_pointer < 32)
                {
                    /* UTF-16 escape sequence uXXXX */
                    escape_characters += 5;
                }
                break;
        }
    }
    output_length = (size_t)(input_pointer - input) + escape_characters;

    output = ensure(output_buffer, output_length + sizeof("\"\""));
    if (output == NULL)
    {
        return false;
    }

    /* no characters have to be escaped */
    if (escape_characters == 0)
    {
        output[0] = '\"';
        memcpy(output + 1, input, output_length);
        output[output_length + 1] = '\"';
        output[output_length + 2] = '\0';

        return true;
    }

    output[0] = '\"';
    output_pointer = output + 1;
    /* copy the string */
    for (input_pointer = input; *input_pointer != '\0'; (void)input_pointer++, output_pointer++)
    {
        if ((*input_pointer > 31) && (*input_pointer != '\"') && (*input_pointer != '\\'))
        {
            /* normal character, copy */
            *output_pointer = *input_pointer;
        }
        else
        {
            /* character needs to be escaped */
            *output_pointer++ = '\\';
            switch (*input_pointer)
            {
                case '\\':
                    *output_pointer = '\\';
                    break;
                case '\"':
                    *output_pointer = '\"';
                    break;
                case '\b':
                    *output_pointer = 'b';
                    break;
                case '\f':
                    *output_pointer = 'f';
                    break;
                case '\n':
                    *output_pointer = 'n';
                    break;
                case '\r':
                    *output_pointer = 'r';
                    break;
                case '\t':
                    *output_pointer = 't';
                    break;
                default:
                    /* escape and print as unicode codepoint */
                    sprintf((char*)output_pointer, "u%04x", *input_pointer);
                    output_pointer += 4;
                    break;
            }
        }
    }
    output[output_length + 1] = '\"';
    output[output_length + 2] = '\0';

    return true;
}

/* Invoke print_string_ptr (which is useful) on an item. */
static cJSON_bool print_string(const cJSON * const item, printbuffer * const p)
{
    return print_string_ptr((unsigned char*)item->valuestring, p);
}

/* Predeclare these prototypes. */
static cJSON_bool parse_value(cJSON * const item, parse_buffer * const input_buffer);
static cJSON_bool print_value(const cJSON * const item, printbuffer * const output_buffer);
static cJSON_bool parse_array(cJSON * const item, parse_buffer * const input_buffer);
static cJSON_bool print_array(const cJSON * const item, printbuffer * const output_buffer);
static cJSON_bool parse_object(cJSON * const item, parse_buffer * const input_buffer);
static cJSON_bool print_object(const cJSON * const item, printbuffer * const output_buffer);

/* Utility to jump whitespace and cr/lf */
static parse_buffer *buffer_skip_whitespace(parse_buffer * const buffer)
{
    if ((buffer == NULL) || (buffer->content == NULL))
    {
        return NULL;
    }

    while (can_access_at_index(buffer, 0) && (buffer_at_offset(buffer)[0] <= 32))
    {
       buffer->offset++;
    }

    if (buffer->offset == buffer->length)
    {
        buffer->offset--;
    }

    return buffer;
}

/* skip the UTF-8 BOM (byte order mark) if it is at the beginning of a buffer */
static parse_buffer *skip_utf8_bom(parse_buffer * const buffer)
{
    if ((buffer == NULL) || (buffer->content == NULL) || (buffer->offset != 0))
    {
        return NULL;
    }

    if (can_access_at_index(buffer, 4) && (strncmp((const char*)buffer_at_offset(buffer), "\xEF\xBB\xBF", 3) == 0))
    {
        buffer->offset += 3;
    }

    return buffer;
}

/* Parse an object - create a new root, and populate. */
CJSON_PUBLIC(cJSON *) cJSON_ParseWithOpts(const char *value, const char **return_parse_end, cJSON_bool require_null_terminated)
{
    parse_buffer buffer = { 0, 0, 0, 0, { 0, 0, 0 } };
    cJSON *item = NULL;

    /* reset error position */
    global_error.json = NULL;
    global_error.position = 0;

    if (value == NULL)
    {
        goto fail;
    }

    buffer.content = (const unsigned char*)value;
    buffer.length = strlen((const char*)value) + sizeof("");
    buffer.offset = 0;
    buffer.hooks = global_hooks;

    item = cJSON_New_Item(&global_hooks);
    if (item == NULL) /* memory fail */
    {
        goto fail;
    }

    if (!parse_value(item, buffer_skip_whitespace(skip_utf8_bom(&buffer))))
    {
        /* parse failure. ep is set. */
        goto fail;
    }

    /* if we require null-terminated JSON without appended garbage, skip and then check for a null terminator */
    if (require_null_terminated)
    {
        buffer_skip_whitespace(&buffer);
        if ((buffer.offset >= buffer.length) || buffer_at_offset(&buffer)[0] != '\0')
        {
            goto fail;
        }
    }
    if (return_parse_end)
    {
        *return_parse_end = (const char*)buffer_at_offset(&buffer);
    }

    return item;

fail:
    if (item != NULL)
    {
        cJSON_Delete(item);
    }

    if (value != NULL)
    {
        error local_error;
        local_error.json = (const unsigned char*)value;
        local_error.position = 0;

        if (buffer.offset < buffer.length)
        {
            local_error.position = buffer.offset;
        }
        else if (buffer.length > 0)
        {
            local_error.position = buffer.length - 1;
        }

        if (return_parse_end != NULL)
        {
            *return_parse_end = (const char*)local_error.json + local_error.position;
        }

        global_error = local_error;
    }

    return NULL;
}

/* Default options for cJSON_Parse */
CJSON_PUBLIC(cJSON *) cJSON_Parse(const char *value)
{
    return cJSON_ParseWithOpts(value, 0, 0);
}

#define cjson_min(a, b) ((a < b) ? a : b)

static unsigned char *print(const cJSON * const item, cJSON_bool format, const internal_hooks * const hooks)
{
    static const size_t default_buffer_size = 256;
    printbuffer buffer[1];
    unsigned char *printed = NULL;

    memset(buffer, 0, sizeof(buffer));

    /* create buffer */
    buffer->buffer = (unsigned char*) hooks->allocate(default_buffer_size);
    buffer->length = default_buffer_size;
    buffer->format = format;
    buffer->hooks = *hooks;
    if (buffer->buffer == NULL)
    {
        goto fail;
    }

    /* print the value */
    if (!print_value(item, buffer))
    {
        goto fail;
    }
    update_offset(buffer);

    /* check if reallocate is available */
    if (hooks->reallocate != NULL)
    {
        printed = (unsigned char*) hooks->reallocate(buffer->buffer, buffer->offset + 1);
        if (printed == NULL) {
            goto fail;
        }
        buffer->buffer = NULL;
    }
    else /* otherwise copy the JSON over to a new buffer */
    {
        printed = (unsigned char*) hooks->allocate(buffer->offset + 1);
        if (printed == NULL)
        {
            goto fail;
        }
        memcpy(printed, buffer->buffer, cjson_min(buffer->length, buffer->offset + 1));
        printed[buffer->offset] = '\0'; /* just to be sure */

        /* free the buffer */
        hooks->deallocate(buffer->buffer);
    }

    return printed;

fail:
    if (buffer->buffer != NULL)
    {
        hooks->deallocate(buffer->buffer);
    }

    if (printed != NULL)
    {
        hooks->deallocate(printed);
    }

    return NULL;
}

/* Render a cJSON item/entity/structure to text. */
CJSON_PUBLIC(char *) cJSON_Print(const cJSON *item)
{
    return (char*)print(item, true, &global_hooks);
}

CJSON_PUBLIC(char *) cJSON_PrintUnformatted(const cJSON *item)
{
    return (char*)print(item, false, &global_hooks);
}

CJSON_PUBLIC(char *) cJSON_PrintBuffered(const cJSON *item, int prebuffer, cJSON_bool fmt)
{
    printbuffer p = { 0, 0, 0, 0, 0, 0, { 0, 0, 0 } };

    if (prebuffer < 0)
    {
        return NULL;
    }

    p.buffer = (unsigned char*)global_hooks.allocate((size_t)prebuffer);
    if (!p.buffer)
    {
        return NULL;
    }

    p.length = (size_t)prebuffer;
    p.offset = 0;
    p.noalloc = false;
    p.format = fmt;
    p.hooks = global_hooks;

    if (!print_value(item, &p))
    {
        global_hooks.deallocate(p.buffer);
        return NULL;
    }

    return (char*)p.buffer;
}

CJSON_PUBLIC(cJSON_bool) cJSON_PrintPreallocated(cJSON *item, char *buf, const int len, const cJSON_bool fmt)
{
    printbuffer p = { 0, 0, 0, 0, 0, 0, { 0, 0, 0 } };

    if ((len < 0) || (buf == NULL))
    {
        return false;
    }

    p.buffer = (unsigned char*)buf;
    p.length = (size_t)len;
    p.offset = 0;
    p.noalloc = true;
    p.format = fmt;
    p.hooks = global_hooks;

    return print_value(item, &p);
}

/* Parser core - when encountering text, process appropriately. */
static cJSON_bool parse_value(cJSON * const item, parse_buffer * const input_buffer)
{
    if ((input_buffer == NULL) || (input_buffer->content == NULL))
    {
        return false; /* no input */
    }

    /* parse the different types of values */
    /* null */
    if (can_read(input_buffer, 4) && (strncmp((const char*)buffer_at_offset(input_buffer), "null", 4) == 0))
    {
        item->type = cJSON_NULL;
        input_buffer->offset += 4;
        return true;
    }
    /* false */
    if (can_read(input_buffer, 5) && (strncmp((const char*)buffer_at_offset(input_buffer), "false", 5) == 0))
    {
        item->type = cJSON_False;
        input_buffer->offset += 5;
        return true;
    }
    /* true */
    if (can_read(input_buffer, 4) && (strncmp((const char*)buffer_at_offset(input_buffer), "true", 4) == 0))
    {
        item->type = cJSON_True;
        item->valueint = 1;
        input_buffer->offset += 4;
        return true;
    }
    /* string */
    if (can_access_at_index(input_buffer, 0) && (buffer_at_offset(input_buffer)[0] == '\"'))
    {
        return parse_string(item, input_buffer);
    }
    /* number */
    if (can_access_at_index(input_buffer, 0) && ((buffer_at_offset(input_buffer)[0] == '-') || ((buffer_at_offset(input_buffer)[0] >= '0') && (buffer_at_offset(input_buffer)[0] <= '9'))))
    {
        return parse_number(item, input_buffer);
    }
    /* array */
    if (can_access_at_index(input_buffer, 0) && (buffer_at_offset(input_buffer)[0] == '['))
    {
        return parse_array(item, input_buffer);
    }
    /* object */
    if (can_access_at_index(input_buffer, 0) && (buffer_at_offset(input_buffer)[0] == '{'))
    {
        return parse_object(item, input_buffer);
    }

    return false;
}

/* Render a value to text. */
static cJSON_bool print_value(const cJSON * const item, printbuffer * const output_buffer)
{
    unsigned char *output = NULL;

    if ((item == NULL) || (output_buffer == NULL))
    {
        return false;
    }

    switch ((item->type) & 0xFF)
    {
        case cJSON_NULL:
            output = ensure(output_buffer, 5);
            if (output == NULL)
            {
                return false;
            }
            strcpy((char*)output, "null");
            return true;

        case cJSON_False:
            output = ensure(output_buffer, 6);
            if (output == NULL)
            {
                return false;
            }
            strcpy((char*)output, "false");
            return true;

        case cJSON_True:
            output = ensure(output_buffer, 5);
            if (output == NULL)
            {
                return false;
            }
            strcpy((char*)output, "true");
            return true;

        case cJSON_Number:
            return print_number(item, output_buffer);

        case cJSON_Raw:
        {
            size_t raw_length = 0;
            if (item->valuestring == NULL)
            {
                return false;
            }

            raw_length = strlen(item->valuestring) + sizeof("");
            output = ensure(output_buffer, raw_length);
            if (output == NULL)
            {
                return false;
            }
            memcpy(output, item->valuestring, raw_length);
            return true;
        }

        case cJSON_String:
            return print_string(item, output_buffer);

        case cJSON_Array:
            return print_array(item, output_buffer);

        case cJSON_Object:
            return print_object(item, output_buffer);

        default:
            return false;
    }
}

/* Build an array from input text. */
static cJSON_bool parse_array(cJSON * const item, parse_buffer * const input_buffer)
{
    cJSON *head = NULL; /* head of the linked list */
    cJSON *current_item = NULL;

    if (input_buffer->depth >= CJSON_NESTING_LIMIT)
    {
        return false; /* to deeply nested */
    }
    input_buffer->depth++;

    if (buffer_at_offset(input_buffer)[0] != '[')
    {
        /* not an array */
        goto fail;
    }

    input_buffer->offset++;
    buffer_skip_whitespace(input_buffer);
    if (can_access_at_index(input_buffer, 0) && (buffer_at_offset(input_buffer)[0] == ']'))
    {
        /* empty array */
        goto success;
    }

    /* check if we skipped to the end of the buffer */
    if (cannot_access_at_index(input_buffer, 0))
    {
        input_buffer->offset--;
        goto fail;
    }

    /* step back to character in front of the first element */
    input_buffer->offset--;
    /* loop through the comma separated array elements */
    do
    {
        /* allocate next item */
        cJSON *new_item = cJSON_New_Item(&(input_buffer->hooks));
        if (new_item == NULL)
        {
            goto fail; /* allocation failure */
        }

        /* attach next item to list */
        if (head == NULL)
        {
            /* start the linked list */
            current_item = head = new_item;
        }
        else
        {
            /* add to the end and advance */
            current_item->next = new_item;
            new_item->prev = current_item;
            current_item = new_item;
        }

        /* parse next value */
        input_buffer->offset++;
        buffer_skip_whitespace(input_buffer);
        if (!parse_value(current_item, input_buffer))
        {
            goto fail; /* failed to parse value */
        }
        buffer_skip_whitespace(input_buffer);
    }
    while (can_access_at_index(input_buffer, 0) && (buffer_at_offset(input_buffer)[0] == ','));

    if (cannot_access_at_index(input_buffer, 0) || buffer_at_offset(input_buffer)[0] != ']')
    {
        goto fail; /* expected end of array */
    }

success:
    input_buffer->depth--;

    item->type = cJSON_Array;
    item->child = head;

    input_buffer->offset++;

    return true;

fail:
    if (head != NULL)
    {
        cJSON_Delete(head);
    }

    return false;
}

/* Render an array to text */
static cJSON_bool print_array(const cJSON * const item, printbuffer * const output_buffer)
{
    unsigned char *output_pointer = NULL;
    size_t length = 0;
    cJSON *current_element = item->child;

    if (output_buffer == NULL)
    {
        return false;
    }

    /* Compose the output array. */
    /* opening square bracket */
    output_pointer = ensure(output_buffer, 1);
    if (output_pointer == NULL)
    {
        return false;
    }

    *output_pointer = '[';
    output_buffer->offset++;
    output_buffer->depth++;

    while (current_element != NULL)
    {
        if (!print_value(current_element, output_buffer))
        {
            return false;
        }
        update_offset(output_buffer);
        if (current_element->next)
        {
            length = (size_t) (output_buffer->format ? 2 : 1);
            output_pointer = ensure(output_buffer, length + 1);
            if (output_pointer == NULL)
            {
                return false;
            }
            *output_pointer++ = ',';
            if(output_buffer->format)
            {
                *output_pointer++ = ' ';
            }
            *output_pointer = '\0';
            output_buffer->offset += length;
        }
        current_element = current_element->next;
    }

    output_pointer = ensure(output_buffer, 2);
    if (output_pointer == NULL)
    {
        return false;
    }
    *output_pointer++ = ']';
    *output_pointer = '\0';
    output_buffer->depth--;

    return true;
}

/* Build an object from the text. */
static cJSON_bool parse_object(cJSON * const item, parse_buffer * const input_buffer)
{
    cJSON *head = NULL; /* linked list head */
    cJSON *current_item = NULL;

    if (input_buffer->depth >= CJSON_NESTING_LIMIT)
    {
        return false; /* to deeply nested */
    }
    input_buffer->depth++;

    if (cannot_access_at_index(input_buffer, 0) || (buffer_at_offset(input_buffer)[0] != '{'))
    {
        goto fail; /* not an object */
    }

    input_buffer->offset++;
    buffer_skip_whitespace(input_buffer);
    if (can_access_at_index(input_buffer, 0) && (buffer_at_offset(input_buffer)[0] == '}'))
    {
        goto success; /* empty object */
    }

    /* check if we skipped to the end of the buffer */
    if (cannot_access_at_index(input_buffer, 0))
    {
        input_buffer->offset--;
        goto fail;
    }

    /* step back to character in front of the first element */
    input_buffer->offset--;
    /* loop through the comma separated array elements */
    do
    {
        /* allocate next item */
        cJSON *new_item = cJSON_New_Item(&(input_buffer->hooks));
        if (new_item == NULL)
        {
            goto fail; /* allocation failure */
        }

        /* attach next item to list */
        if (head == NULL)
        {
            /* start the linked list */
            current_item = head = new_item;
        }
        else
        {
            /* add to the end and advance */
            current_item->next = new_item;
            new_item->prev = current_item;
            current_item = new_item;
        }

        /* parse the name of the child */
        input_buffer->offset++;
        buffer_skip_whitespace(input_buffer);
        if (!parse_string(current_item, input_buffer))
        {
            goto fail; /* faile to parse name */
        }
        buffer_skip_whitespace(input_buffer);

        /* swap valuestring and string, because we parsed the name */
        current_item->string = current_item->valuestring;
        current_item->valuestring = NULL;

        if (cannot_access_at_index(input_buffer, 0) || (buffer_at_offset(input_buffer)[0] != ':'))
        {
            goto fail; /* invalid object */
        }

        /* parse the value */
        input_buffer->offset++;
        buffer_skip_whitespace(input_buffer);
        if (!parse_value(current_item, input_buffer))
        {
            goto fail; /* failed to parse value */
        }
        buffer_skip_whitespace(input_buffer);
    }
    while (can_access_at_index(input_buffer, 0) && (buffer_at_offset(input_buffer)[0] == ','));

    if (cannot_access_at_index(input_buffer, 0) || (buffer_at_offset(input_buffer)[0] != '}'))
    {
        goto fail; /* expected end of object */
    }

success:
    input_buffer->depth--;

    item->type = cJSON_Object;
    item->child = head;

    input_buffer->offset++;
    return true;

fail:
    if (head != NULL)
    {
        cJSON_Delete(head);
    }

    return false;
}

/* Render an object to text. */
static cJSON_bool print_object(const cJSON * const item, printbuffer * const output_buffer)
{
    unsigned char *output_pointer = NULL;
    size_t length = 0;
    cJSON *current_item = item->child;

    if (output_buffer == NULL)
    {
        return false;
    }

    /* Compose the output: */
    length = (size_t) (output_buffer->format ? 2 : 1); /* fmt: {\n */
    output_pointer = ensure(output_buffer, length + 1);
    if (output_pointer == NULL)
    {
        return false;
    }

    *output_pointer++ = '{';
    output_buffer->depth++;
    if (output_buffer->format)
    {
        *output_pointer++ = '\n';
    }
    output_buffer->offset += length;

    while (current_item)
    {
        if (output_buffer->format)
        {
            size_t i;
            output_pointer = ensure(output_buffer, output_buffer->depth);
            if (output_pointer == NULL)
            {
                return false;
            }
            for (i = 0; i < output_buffer->depth; i++)
            {
                *output_pointer++ = '\t';
            }
            output_buffer->offset += output_buffer->depth;
        }

        /* print key */
        if (!print_string_ptr((unsigned char*)current_item->string, output_buffer))
        {
            return false;
        }
        update_offset(output_buffer);

        length = (size_t) (output_buffer->format ? 2 : 1);
        output_pointer = ensure(output_buffer, length);
        if (output_pointer == NULL)
        {
            return false;
        }
        *output_pointer++ = ':';
        if (output_buffer->format)
        {
            *output_pointer++ = '\t';
        }
        output_buffer->offset += length;

        /* print value */
        if (!print_value(current_item, output_buffer))
        {
            return false;
        }
        update_offset(output_buffer);

        /* print comma if not last */
        length = (size_t) ((output_buffer->format ? 1 : 0) + (current_item->next ? 1 : 0));
        output_pointer = ensure(output_buffer, length + 1);
        if (output_pointer == NULL)
        {
            return false;
        }
        if (current_item->next)
        {
            *output_pointer++ = ',';
        }

        if (output_buffer->format)
        {
            *output_pointer++ = '\n';
        }
        *output_pointer = '\0';
        output_buffer->offset += length;

        current_item = current_item->next;
    }

    output_pointer = ensure(output_buffer, output_buffer->format ? (output_buffer->depth + 1) : 2);
    if (output_pointer == NULL)
    {
        return false;
    }
    if (output_buffer->format)
    {
        size_t i;
        for (i = 0; i < (output_buffer->depth - 1); i++)
        {
            *output_pointer++ = '\t';
        }
    }
    *output_pointer++ = '}';
    *output_pointer = '\0';
    output_buffer->depth--;

    return true;
}

/* Get Array size/item / object item. */
CJSON_PUBLIC(int) cJSON_GetArraySize(const cJSON *array)
{
    cJSON *child = NULL;
    size_t size = 0;

    if (array == NULL)
    {
        return 0;
    }

    child = array->child;

    while(child != NULL)
    {
        size++;
        child = child->next;
    }

    /* FIXME: Can overflow here. Cannot be fixed without breaking the API */

    return (int)size;
}

static cJSON* get_array_item(const cJSON *array, size_t index)
{
    cJSON *current_child = NULL;

    if (array == NULL)
    {
        return NULL;
    }

    current_child = array->child;
    while ((current_child != NULL) && (index > 0))
    {
        index--;
        current_child = current_child->next;
    }

    return current_child;
}

CJSON_PUBLIC(cJSON *) cJSON_GetArrayItem(const cJSON *array, int index)
{
    if (index < 0)
    {
        return NULL;
    }

    return get_array_item(array, (size_t)index);
}

static cJSON *get_object_item(const cJSON * const object, const char * const name, const cJSON_bool case_sensitive)
{
    cJSON *current_element = NULL;

    if ((object == NULL) || (name == NULL))
    {
        return NULL;
    }

    current_element = object->child;
    if (case_sensitive)
    {
        while ((current_element != NULL) && (strcmp(name, current_element->string) != 0))
        {
            current_element = current_element->next;
        }
    }
    else
    {
        while ((current_element != NULL) && (case_insensitive_strcmp((const unsigned char*)name, (const unsigned char*)(current_element->string)) != 0))
        {
            current_element = current_element->next;
        }
    }

    return current_element;
}

CJSON_PUBLIC(cJSON *) cJSON_GetObjectItem(const cJSON * const object, const char * const string)
{
    return get_object_item(object, string, false);
}

CJSON_PUBLIC(cJSON *) cJSON_GetObjectItemCaseSensitive(const cJSON * const object, const char * const string)
{
    return get_object_item(object, string, true);
}

CJSON_PUBLIC(cJSON_bool) cJSON_HasObjectItem(const cJSON *object, const char *string)
{
    return cJSON_GetObjectItem(object, string) ? 1 : 0;
}

/* Utility for array list handling. */
static void suffix_object(cJSON *prev, cJSON *item)
{
    prev->next = item;
    item->prev = prev;
}

/* Utility for handling references. */
static cJSON *create_reference(const cJSON *item, const internal_hooks * const hooks)
{
    cJSON *reference = NULL;
    if (item == NULL)
    {
        return NULL;
    }

    reference = cJSON_New_Item(hooks);
    if (reference == NULL)
    {
        return NULL;
    }

    memcpy(reference, item, sizeof(cJSON));
    reference->string = NULL;
    reference->type |= cJSON_IsReference;
    reference->next = reference->prev = NULL;
    return reference;
}

static cJSON_bool add_item_to_array(cJSON *array, cJSON *item)
{
    cJSON *child = NULL;

    if ((item == NULL) || (array == NULL))
    {
        return false;
    }

    child = array->child;

    if (child == NULL)
    {
        /* list is empty, start new one */
        array->child = item;
    }
    else
    {
        /* append to the end */
        while (child->next)
        {
            child = child->next;
        }
        suffix_object(child, item);
    }

    return true;
}

/* Add item to array/object. */
CJSON_PUBLIC(void) cJSON_AddItemToArray(cJSON *array, cJSON *item)
{
    add_item_to_array(array, item);
}

#if defined(__clang__) || (defined(__GNUC__)  && ((__GNUC__ > 4) || ((__GNUC__ == 4) && (__GNUC_MINOR__ > 5))))
    #pragma GCC diagnostic push
#endif
#ifdef __GNUC__
#pragma GCC diagnostic ignored "-Wcast-qual"
#endif
/* helper function to cast away const */
static void* cast_away_const(const void* string)
{
    return (void*)string;
}
#if defined(__clang__) || (defined(__GNUC__)  && ((__GNUC__ > 4) || ((__GNUC__ == 4) && (__GNUC_MINOR__ > 5))))
    #pragma GCC diagnostic pop
#endif


static cJSON_bool add_item_to_object(cJSON * const object, const char * const string, cJSON * const item, const internal_hooks * const hooks, const cJSON_bool constant_key)
{
    char *new_key = NULL;
    int new_type = cJSON_Invalid;

    if ((object == NULL) || (string == NULL) || (item == NULL))
    {
        return false;
    }

    if (constant_key)
    {
        new_key = (char*)cast_away_const(string);
        new_type = item->type | cJSON_StringIsConst;
    }
    else
    {
        new_key = (char*)cJSON_strdup((const unsigned char*)string, hooks);
        if (new_key == NULL)
        {
            return false;
        }

        new_type = item->type & ~cJSON_StringIsConst;
    }

    if (!(item->type & cJSON_StringIsConst) && (item->string != NULL))
    {
        hooks->deallocate(item->string);
    }

    item->string = new_key;
    item->type = new_type;

    return add_item_to_array(object, item);
}

CJSON_PUBLIC(void) cJSON_AddItemToObject(cJSON *object, const char *string, cJSON *item)
{
    add_item_to_object(object, string, item, &global_hooks, false);
}

/* Add an item to an object with constant string as key */
CJSON_PUBLIC(void) cJSON_AddItemToObjectCS(cJSON *object, const char *string, cJSON *item)
{
    add_item_to_object(object, string, item, &global_hooks, true);
}

CJSON_PUBLIC(void) cJSON_AddItemReferenceToArray(cJSON *array, cJSON *item)
{
    if (array == NULL)
    {
        return;
    }

    add_item_to_array(array, create_reference(item, &global_hooks));
}

CJSON_PUBLIC(void) cJSON_AddItemReferenceToObject(cJSON *object, const char *string, cJSON *item)
{
    if ((object == NULL) || (string == NULL))
    {
        return;
    }

    add_item_to_object(object, string, create_reference(item, &global_hooks), &global_hooks, false);
}

CJSON_PUBLIC(cJSON*) cJSON_AddNullToObject(cJSON * const object, const char * const name)
{
    cJSON *null = cJSON_CreateNull();
    if (add_item_to_object(object, name, null, &global_hooks, false))
    {
        return null;
    }

    cJSON_Delete(null);
    return NULL;
}

CJSON_PUBLIC(cJSON*) cJSON_AddTrueToObject(cJSON * const object, const char * const name)
{
    cJSON *true_item = cJSON_CreateTrue();
    if (add_item_to_object(object, name, true_item, &global_hooks, false))
    {
        return true_item;
    }

    cJSON_Delete(true_item);
    return NULL;
}

CJSON_PUBLIC(cJSON*) cJSON_AddFalseToObject(cJSON * const object, const char * const name)
{
    cJSON *false_item = cJSON_CreateFalse();
    if (add_item_to_object(object, name, false_item, &global_hooks, false))
    {
        return false_item;
    }

    cJSON_Delete(false_item);
    return NULL;
}

CJSON_PUBLIC(cJSON*) cJSON_AddBoolToObject(cJSON * const object, const char * const name, const cJSON_bool boolean)
{
    cJSON *bool_item = cJSON_CreateBool(boolean);
    if (add_item_to_object(object, name, bool_item, &global_hooks, false))
    {
        return bool_item;
    }

    cJSON_Delete(bool_item);
    return NULL;
}

CJSON_PUBLIC(cJSON*) cJSON_AddNumberToObject(cJSON * const object, const char * const name, const double number)
{
    cJSON *number_item = cJSON_CreateNumber(number);
    if (add_item_to_object(object, name, number_item, &global_hooks, false))
    {
        return number_item;
    }

    cJSON_Delete(number_item);
    return NULL;
}

CJSON_PUBLIC(cJSON*) cJSON_AddStringToObject(cJSON * const object, const char * const name, const char * const string)
{
    cJSON *string_item = cJSON_CreateString(string);
    if (add_item_to_object(object, name, string_item, &global_hooks, false))
    {
        return string_item;
    }

    cJSON_Delete(string_item);
    return NULL;
}

CJSON_PUBLIC(cJSON*) cJSON_AddRawToObject(cJSON * const object, const char * const name, const char * const raw)
{
    cJSON *raw_item = cJSON_CreateRaw(raw);
    if (add_item_to_object(object, name, raw_item, &global_hooks, false))
    {
        return raw_item;
    }

    cJSON_Delete(raw_item);
    return NULL;
}

CJSON_PUBLIC(cJSON*) cJSON_AddObjectToObject(cJSON * const object, const char * const name)
{
    cJSON *object_item = cJSON_CreateObject();
    if (add_item_to_object(object, name, object_item, &global_hooks, false))
    {
        return object_item;
    }

    cJSON_Delete(object_item);
    return NULL;
}

CJSON_PUBLIC(cJSON*) cJSON_AddArrayToObject(cJSON * const object, const char * const name)
{
    cJSON *array = cJSON_CreateArray();
    if (add_item_to_object(object, name, array, &global_hooks, false))
    {
        return array;
    }

    cJSON_Delete(array);
    return NULL;
}

CJSON_PUBLIC(cJSON *) cJSON_DetachItemViaPointer(cJSON *parent, cJSON * const item)
{
    if ((parent == NULL) || (item == NULL))
    {
        return NULL;
    }

    if (item->prev != NULL)
    {
        /* not the first element */
        item->prev->next = item->next;
    }
    if (item->next != NULL)
    {
        /* not the last element */
        item->next->prev = item->prev;
    }

    if (item == parent->child)
    {
        /* first element */
        parent->child = item->next;
    }
    /* make sure the detached item doesn't point anywhere anymore */
    item->prev = NULL;
    item->next = NULL;

    return item;
}

CJSON_PUBLIC(cJSON *) cJSON_DetachItemFromArray(cJSON *array, int which)
{
    if (which < 0)
    {
        return NULL;
    }

    return cJSON_DetachItemViaPointer(array, get_array_item(array, (size_t)which));
}

CJSON_PUBLIC(void) cJSON_DeleteItemFromArray(cJSON *array, int which)
{
    cJSON_Delete(cJSON_DetachItemFromArray(array, which));
}

CJSON_PUBLIC(cJSON *) cJSON_DetachItemFromObject(cJSON *object, const char *string)
{
    cJSON *to_detach = cJSON_GetObjectItem(object, string);

    return cJSON_DetachItemViaPointer(object, to_detach);
}

CJSON_PUBLIC(cJSON *) cJSON_DetachItemFromObjectCaseSensitive(cJSON *object, const char *string)
{
    cJSON *to_detach = cJSON_GetObjectItemCaseSensitive(object, string);

    return cJSON_DetachItemViaPointer(object, to_detach);
}

CJSON_PUBLIC(void) cJSON_DeleteItemFromObject(cJSON *object, const char *string)
{
    cJSON_Delete(cJSON_DetachItemFromObject(object, string));
}

CJSON_PUBLIC(void) cJSON_DeleteItemFromObjectCaseSensitive(cJSON *object, const char *string)
{
    cJSON_Delete(cJSON_DetachItemFromObjectCaseSensitive(object, string));
}

/* Replace array/object items with new ones. */
CJSON_PUBLIC(void) cJSON_InsertItemInArray(cJSON *array, int which, cJSON *newitem)
{
    cJSON *after_inserted = NULL;

    if (which < 0)
    {
        return;
    }

    after_inserted = get_array_item(array, (size_t)which);
    if (after_inserted == NULL)
    {
        add_item_to_array(array, newitem);
        return;
    }

    newitem->next = after_inserted;
    newitem->prev = after_inserted->prev;
    after_inserted->prev = newitem;
    if (after_inserted == array->child)
    {
        array->child = newitem;
    }
    else
    {
        newitem->prev->next = newitem;
    }
}

CJSON_PUBLIC(cJSON_bool) cJSON_ReplaceItemViaPointer(cJSON * const parent, cJSON * const item, cJSON * replacement)
{
    if ((parent == NULL) || (replacement == NULL) || (item == NULL))
    {
        return false;
    }

    if (replacement == item)
    {
        return true;
    }

    replacement->next = item->next;
    replacement->prev = item->prev;

    if (replacement->next != NULL)
    {
        replacement->next->prev = replacement;
    }
    if (replacement->prev != NULL)
    {
        replacement->prev->next = replacement;
    }
    if (parent->child == item)
    {
        parent->child = replacement;
    }

    item->next = NULL;
    item->prev = NULL;
    cJSON_Delete(item);

    return true;
}

CJSON_PUBLIC(void) cJSON_ReplaceItemInArray(cJSON *array, int which, cJSON *newitem)
{
    if (which < 0)
    {
        return;
    }

    cJSON_ReplaceItemViaPointer(array, get_array_item(array, (size_t)which), newitem);
}

static cJSON_bool replace_item_in_object(cJSON *object, const char *string, cJSON *replacement, cJSON_bool case_sensitive)
{
    if ((replacement == NULL) || (string == NULL))
    {
        return false;
    }

    /* replace the name in the replacement */
    if (!(replacement->type & cJSON_StringIsConst) && (replacement->string != NULL))
    {
        cJSON_free(replacement->string);
    }
    replacement->string = (char*)cJSON_strdup((const unsigned char*)string, &global_hooks);
    replacement->type &= ~cJSON_StringIsConst;

    cJSON_ReplaceItemViaPointer(object, get_object_item(object, string, case_sensitive), replacement);

    return true;
}

CJSON_PUBLIC(void) cJSON_ReplaceItemInObject(cJSON *object, const char *string, cJSON *newitem)
{
    replace_item_in_object(object, string, newitem, false);
}

CJSON_PUBLIC(void) cJSON_ReplaceItemInObjectCaseSensitive(cJSON *object, const char *string, cJSON *newitem)
{
    replace_item_in_object(object, string, newitem, true);
}

/* Create basic types: */
CJSON_PUBLIC(cJSON *) cJSON_CreateNull(void)
{
    cJSON *item = cJSON_New_Item(&global_hooks);
    if(item)
    {
        item->type = cJSON_NULL;
    }

    return item;
}

CJSON_PUBLIC(cJSON *) cJSON_CreateTrue(void)
{
    cJSON *item = cJSON_New_Item(&global_hooks);
    if(item)
    {
        item->type = cJSON_True;
    }

    return item;
}

CJSON_PUBLIC(cJSON *) cJSON_CreateFalse(void)
{
    cJSON *item = cJSON_New_Item(&global_hooks);
    if(item)
    {
        item->type = cJSON_False;
    }

    return item;
}

CJSON_PUBLIC(cJSON *) cJSON_CreateBool(cJSON_bool b)
{
    cJSON *item = cJSON_New_Item(&global_hooks);
    if(item)
    {
        item->type = b ? cJSON_True : cJSON_False;
    }

    return item;
}

CJSON_PUBLIC(cJSON *) cJSON_CreateNumber(double num)
{
    cJSON *item = cJSON_New_Item(&global_hooks);
    if(item)
    {
        item->type = cJSON_Number;
        item->valuedouble = num;

        /* use saturation in case of overflow */
        if (num >= INT_MAX)
        {
            item->valueint = INT_MAX;
        }
        else if (num <= INT_MIN)
        {
            item->valueint = INT_MIN;
        }
        else
        {
            item->valueint = (int)num;
        }
    }

    return item;
}

CJSON_PUBLIC(cJSON *) cJSON_CreateString(const char *string)
{
    cJSON *item = cJSON_New_Item(&global_hooks);
    if(item)
    {
        item->type = cJSON_String;
        item->valuestring = (char*)cJSON_strdup((const unsigned char*)string, &global_hooks);
        if(!item->valuestring)
        {
            cJSON_Delete(item);
            return NULL;
        }
    }

    return item;
}

CJSON_PUBLIC(cJSON *) cJSON_CreateStringReference(const char *string)
{
    cJSON *item = cJSON_New_Item(&global_hooks);
    if (item != NULL)
    {
        item->type = cJSON_String | cJSON_IsReference;
        item->valuestring = (char*)cast_away_const(string);
    }

    return item;
}

CJSON_PUBLIC(cJSON *) cJSON_CreateObjectReference(const cJSON *child)
{
    cJSON *item = cJSON_New_Item(&global_hooks);
    if (item != NULL) {
        item->type = cJSON_Object | cJSON_IsReference;
        item->child = (cJSON*)cast_away_const(child);
    }

    return item;
}

CJSON_PUBLIC(cJSON *) cJSON_CreateArrayReference(const cJSON *child) {
    cJSON *item = cJSON_New_Item(&global_hooks);
    if (item != NULL) {
        item->type = cJSON_Array | cJSON_IsReference;
        item->child = (cJSON*)cast_away_const(child);
    }

    return item;
}

CJSON_PUBLIC(cJSON *) cJSON_CreateRaw(const char *raw)
{
    cJSON *item = cJSON_New_Item(&global_hooks);
    if(item)
    {
        item->type = cJSON_Raw;
        item->valuestring = (char*)cJSON_strdup((const unsigned char*)raw, &global_hooks);
        if(!item->valuestring)
        {
            cJSON_Delete(item);
            return NULL;
        }
    }

    return item;
}

CJSON_PUBLIC(cJSON *) cJSON_CreateArray(void)
{
    cJSON *item = cJSON_New_Item(&global_hooks);
    if(item)
    {
        item->type=cJSON_Array;
    }

    return item;
}

CJSON_PUBLIC(cJSON *) cJSON_CreateObject(void)
{
    cJSON *item = cJSON_New_Item(&global_hooks);
    if (item)
    {
        item->type = cJSON_Object;
    }

    return item;
}

/* Create Arrays: */
CJSON_PUBLIC(cJSON *) cJSON_CreateIntArray(const int *numbers, int count)
{
    size_t i = 0;
    cJSON *n = NULL;
    cJSON *p = NULL;
    cJSON *a = NULL;

    if ((count < 0) || (numbers == NULL))
    {
        return NULL;
    }

    a = cJSON_CreateArray();
    for(i = 0; a && (i < (size_t)count); i++)
    {
        n = cJSON_CreateNumber(numbers[i]);
        if (!n)
        {
            cJSON_Delete(a);
            return NULL;
        }
        if(!i)
        {
            a->child = n;
        }
        else
        {
            suffix_object(p, n);
        }
        p = n;
    }

    return a;
}

CJSON_PUBLIC(cJSON *) cJSON_CreateFloatArray(const float *numbers, int count)
{
    size_t i = 0;
    cJSON *n = NULL;
    cJSON *p = NULL;
    cJSON *a = NULL;

    if ((count < 0) || (numbers == NULL))
    {
        return NULL;
    }

    a = cJSON_CreateArray();

    for(i = 0; a && (i < (size_t)count); i++)
    {
        n = cJSON_CreateNumber((double)numbers[i]);
        if(!n)
        {
            cJSON_Delete(a);
            return NULL;
        }
        if(!i)
        {
            a->child = n;
        }
        else
        {
            suffix_object(p, n);
        }
        p = n;
    }

    return a;
}

CJSON_PUBLIC(cJSON *) cJSON_CreateDoubleArray(const double *numbers, int count)
{
    size_t i = 0;
    cJSON *n = NULL;
    cJSON *p = NULL;
    cJSON *a = NULL;

    if ((count < 0) || (numbers == NULL))
    {
        return NULL;
    }

    a = cJSON_CreateArray();

    for(i = 0;a && (i < (size_t)count); i++)
    {
        n = cJSON_CreateNumber(numbers[i]);
        if(!n)
        {
            cJSON_Delete(a);
            return NULL;
        }
        if(!i)
        {
            a->child = n;
        }
        else
        {
            suffix_object(p, n);
        }
        p = n;
    }

    return a;
}

CJSON_PUBLIC(cJSON *) cJSON_CreateStringArray(const char **strings, int count)
{
    size_t i = 0;
    cJSON *n = NULL;
    cJSON *p = NULL;
    cJSON *a = NULL;

    if ((count < 0) || (strings == NULL))
    {
        return NULL;
    }

    a = cJSON_CreateArray();

    for (i = 0; a && (i < (size_t)count); i++)
    {
        n = cJSON_CreateString(strings[i]);
        if(!n)
        {
            cJSON_Delete(a);
            return NULL;
        }
        if(!i)
        {
            a->child = n;
        }
        else
        {
            suffix_object(p,n);
        }
        p = n;
    }

    return a;
}

/* Duplication */
CJSON_PUBLIC(cJSON *) cJSON_Duplicate(const cJSON *item, cJSON_bool recurse)
{
    cJSON *newitem = NULL;
    cJSON *child = NULL;
    cJSON *next = NULL;
    cJSON *newchild = NULL;

    /* Bail on bad ptr */
    if (!item)
    {
        goto fail;
    }
    /* Create new item */
    newitem = cJSON_New_Item(&global_hooks);
    if (!newitem)
    {
        goto fail;
    }
    /* Copy over all vars */
    newitem->type = item->type & (~cJSON_IsReference);
    newitem->valueint = item->valueint;
    newitem->valuedouble = item->valuedouble;
    if (item->valuestring)
    {
        newitem->valuestring = (char*)cJSON_strdup((unsigned char*)item->valuestring, &global_hooks);
        if (!newitem->valuestring)
        {
            goto fail;
        }
    }
    if (item->string)
    {
        newitem->string = (item->type&cJSON_StringIsConst) ? item->string : (char*)cJSON_strdup((unsigned char*)item->string, &global_hooks);
        if (!newitem->string)
        {
            goto fail;
        }
    }
    /* If non-recursive, then we're done! */
    if (!recurse)
    {
        return newitem;
    }
    /* Walk the ->next chain for the child. */
    child = item->child;
    while (child != NULL)
    {
        newchild = cJSON_Duplicate(child, true); /* Duplicate (with recurse) each item in the ->next chain */
        if (!newchild)
        {
            goto fail;
        }
        if (next != NULL)
        {
            /* If newitem->child already set, then crosswire ->prev and ->next and move on */
            next->next = newchild;
            newchild->prev = next;
            next = newchild;
        }
        else
        {
            /* Set newitem->child and move to it */
            newitem->child = newchild;
            next = newchild;
        }
        child = child->next;
    }

    return newitem;

fail:
    if (newitem != NULL)
    {
        cJSON_Delete(newitem);
    }

    return NULL;
}

CJSON_PUBLIC(void) cJSON_Minify(char *json)
{
    unsigned char *into = (unsigned char*)json;

    if (json == NULL)
    {
        return;
    }

    while (*json)
    {
        if (*json == ' ')
        {
            json++;
        }
        else if (*json == '\t')
        {
            /* Whitespace characters. */
            json++;
        }
        else if (*json == '\r')
        {
            json++;
        }
        else if (*json=='\n')
        {
            json++;
        }
        else if ((*json == '/') && (json[1] == '/'))
        {
            /* double-slash comments, to end of line. */
            while (*json && (*json != '\n'))
            {
                json++;
            }
        }
        else if ((*json == '/') && (json[1] == '*'))
        {
            /* multiline comments. */
            while (*json && !((*json == '*') && (json[1] == '/')))
            {
                json++;
            }
            json += 2;
        }
        else if (*json == '\"')
        {
            /* string literals, which are \" sensitive. */
            *into++ = (unsigned char)*json++;
            while (*json && (*json != '\"'))
            {
                if (*json == '\\')
                {
                    *into++ = (unsigned char)*json++;
                }
                *into++ = (unsigned char)*json++;
            }
            *into++ = (unsigned char)*json++;
        }
        else
        {
            /* All other characters. */
            *into++ = (unsigned char)*json++;
        }
    }

    /* and null-terminate. */
    *into = '\0';
}

CJSON_PUBLIC(cJSON_bool) cJSON_IsInvalid(const cJSON * const item)
{
    if (item == NULL)
    {
        return false;
    }

    return (item->type & 0xFF) == cJSON_Invalid;
}

CJSON_PUBLIC(cJSON_bool) cJSON_IsFalse(const cJSON * const item)
{
    if (item == NULL)
    {
        return false;
    }

    return (item->type & 0xFF) == cJSON_False;
}

CJSON_PUBLIC(cJSON_bool) cJSON_IsTrue(const cJSON * const item)
{
    if (item == NULL)
    {
        return false;
    }

    return (item->type & 0xff) == cJSON_True;
}


CJSON_PUBLIC(cJSON_bool) cJSON_IsBool(const cJSON * const item)
{
    if (item == NULL)
    {
        return false;
    }

    return (item->type & (cJSON_True | cJSON_False)) != 0;
}
CJSON_PUBLIC(cJSON_bool) cJSON_IsNull(const cJSON * const item)
{
    if (item == NULL)
    {
        return false;
    }

    return (item->type & 0xFF) == cJSON_NULL;
}

CJSON_PUBLIC(cJSON_bool) cJSON_IsNumber(const cJSON * const item)
{
    if (item == NULL)
    {
        return false;
    }

    return (item->type & 0xFF) == cJSON_Number;
}

CJSON_PUBLIC(cJSON_bool) cJSON_IsString(const cJSON * const item)
{
    if (item == NULL)
    {
        return false;
    }

    return (item->type & 0xFF) == cJSON_String;
}

CJSON_PUBLIC(cJSON_bool) cJSON_IsArray(const cJSON * const item)
{
    if (item == NULL)
    {
        return false;
    }

    return (item->type & 0xFF) == cJSON_Array;
}

CJSON_PUBLIC(cJSON_bool) cJSON_IsObject(const cJSON * const item)
{
    if (item == NULL)
    {
        return false;
    }

    return (item->type & 0xFF) == cJSON_Object;
}

CJSON_PUBLIC(cJSON_bool) cJSON_IsRaw(const cJSON * const item)
{
    if (item == NULL)
    {
        return false;
    }

    return (item->type & 0xFF) == cJSON_Raw;
}

CJSON_PUBLIC(cJSON_bool) cJSON_Compare(const cJSON * const a, const cJSON * const b, const cJSON_bool case_sensitive)
{
    if ((a == NULL) || (b == NULL) || ((a->type & 0xFF) != (b->type & 0xFF)) || cJSON_IsInvalid(a))
    {
        return false;
    }

    /* check if type is valid */
    switch (a->type & 0xFF)
    {
        case cJSON_False:
        case cJSON_True:
        case cJSON_NULL:
        case cJSON_Number:
        case cJSON_String:
        case cJSON_Raw:
        case cJSON_Array:
        case cJSON_Object:
            break;

        default:
            return false;
    }

    /* identical objects are equal */
    if (a == b)
    {
        return true;
    }

    switch (a->type & 0xFF)
    {
        /* in these cases and equal type is enough */
        case cJSON_False:
        case cJSON_True:
        case cJSON_NULL:
            return true;

        case cJSON_Number:
            if (a->valuedouble == b->valuedouble)
            {
                return true;
            }
            return false;

        case cJSON_String:
        case cJSON_Raw:
            if ((a->valuestring == NULL) || (b->valuestring == NULL))
            {
                return false;
            }
            if (strcmp(a->valuestring, b->valuestring) == 0)
            {
                return true;
            }

            return false;

        case cJSON_Array:
        {
            cJSON *a_element = a->child;
            cJSON *b_element = b->child;

            for (; (a_element != NULL) && (b_element != NULL);)
            {
                if (!cJSON_Compare(a_element, b_element, case_sensitive))
                {
                    return false;
                }

                a_element = a_element->next;
                b_element = b_element->next;
            }

            /* one of the arrays is longer than the other */
            if (a_element != b_element) {
                return false;
            }

            return true;
        }

        case cJSON_Object:
        {
            cJSON *a_element = NULL;
            cJSON *b_element = NULL;
            cJSON_ArrayForEach(a_element, a)
            {
                /* TODO This has O(n^2) runtime, which is horrible! */
                b_element = get_object_item(b, a_element->string, case_sensitive);
                if (b_element == NULL)
                {
                    return false;
                }

                if (!cJSON_Compare(a_element, b_element, case_sensitive))
                {
                    return false;
                }
            }

            /* doing this twice, once on a and b to prevent true comparison if a subset of b
             * TODO: Do this the proper way, this is just a fix for now */
            cJSON_ArrayForEach(b_element, b)
            {
                a_element = get_object_item(a, b_element->string, case_sensitive);
                if (a_element == NULL)
                {
                    return false;
                }

                if (!cJSON_Compare(b_element, a_element, case_sensitive))
                {
                    return false;
                }
            }

            return true;
        }

        default:
            return false;
    }
}

CJSON_PUBLIC(void *) cJSON_malloc(size_t size)
{
    return global_hooks.allocate(size);
}

CJSON_PUBLIC(void) cJSON_free(void *object)
{
    global_hooks.deallocate(object);
}



================================================
FILE: src/cJSON.h
================================================
/*
  Copyright (c) 2009-2017 Dave Gamble and cJSON contributors

  Permission is hereby granted, free of charge, to any person obtaining a copy
  of this software and associated documentation files (the "Software"), to deal
  in the Software without restriction, including without limitation the rights
  to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
  copies of the Software, and to permit persons to whom the Software is
  furnished to do so, subject to the following conditions:

  The above copyright notice and this permission notice shall be included in
  all copies or substantial portions of the Software.

  THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
  IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
  FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
  AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
  LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
  OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
  THE SOFTWARE.
*/

#ifndef cJSON__h
#define cJSON__h

#ifdef __cplusplus
extern "C"
{
#endif

/* project version */
#define CJSON_VERSION_MAJOR 1
#define CJSON_VERSION_MINOR 7
#define CJSON_VERSION_PATCH 7

#include <stddef.h>

/* cJSON Types: */
#define cJSON_Invalid (0)
#define cJSON_False  (1 << 0)
#define cJSON_True   (1 << 1)
#define cJSON_NULL   (1 << 2)
#define cJSON_Number (1 << 3)
#define cJSON_String (1 << 4)
#define cJSON_Array  (1 << 5)
#define cJSON_Object (1 << 6)
#define cJSON_Raw    (1 << 7) /* raw json */

#define cJSON_IsReference 256
#define cJSON_StringIsConst 512

/* The cJSON structure: */
typedef struct cJSON
{
    /* next/prev allow you to walk array/object chains. Alternatively, use GetArraySize/GetArrayItem/GetObjectItem */
    struct cJSON *next;
    struct cJSON *prev;
    /* An array or object item will have a child pointer pointing to a chain of the items in the array/object. */
    struct cJSON *child;

    /* The type of the item, as above. */
    int type;

    /* The item's string, if type==cJSON_String  and type == cJSON_Raw */
    char *valuestring;
    /* writing to valueint is DEPRECATED, use cJSON_SetNumberValue instead */
    int valueint;
    /* The item's number, if type==cJSON_Number */
    double valuedouble;

    /* The item's name string, if this item is the child of, or is in the list of subitems of an object. */
    char *string;
} cJSON;

typedef struct cJSON_Hooks
{
      void *(*malloc_fn)(size_t sz);
      void (*free_fn)(void *ptr);
} cJSON_Hooks;

typedef int cJSON_bool;

#if !defined(__WINDOWS__) && (defined(WIN32) || defined(WIN64) || defined(_MSC_VER) || defined(_WIN32))
#define __WINDOWS__
#endif
#ifdef __WINDOWS__

/* When compiling for windows, we specify a specific calling convention to avoid issues where we are being called from a project with a different default calling convention.  For windows you have 2 define options:

CJSON_HIDE_SYMBOLS - Define this in the case where you don't want to ever dllexport symbols
CJSON_EXPORT_SYMBOLS - Define this on library build when you want to dllexport symbols (default)
CJSON_IMPORT_SYMBOLS - Define this if you want to dllimport symbol

For *nix builds that support visibility attribute, you can define similar behavior by

setting default visibility to hidden by adding
-fvisibility=hidden (for gcc)
or
-xldscope=hidden (for sun cc)
to CFLAGS

then using the CJSON_API_VISIBILITY flag to "export" the same symbols the way CJSON_EXPORT_SYMBOLS does

*/

/* export symbols by default, this is necessary for copy pasting the C and header file */
#if !defined(CJSON_HIDE_SYMBOLS) && !defined(CJSON_IMPORT_SYMBOLS) && !defined(CJSON_EXPORT_SYMBOLS)
#define CJSON_EXPORT_SYMBOLS
#endif

#if defined(CJSON_HIDE_SYMBOLS)
#define CJSON_PUBLIC(type)   type __stdcall
#elif defined(CJSON_EXPORT_SYMBOLS)
#define CJSON_PUBLIC(type)   __declspec(dllexport) type __stdcall
#elif defined(CJSON_IMPORT_SYMBOLS)
#define CJSON_PUBLIC(type)   __declspec(dllimport) type __stdcall
#endif
#else /* !WIN32 */
#if (defined(__GNUC__) || defined(__SUNPRO_CC) || defined (__SUNPRO_C)) && defined(CJSON_API_VISIBILITY)
#define CJSON_PUBLIC(type)   __attribute__((visibility("default"))) type
#else
#define CJSON_PUBLIC(type) type
#endif
#endif

/* Limits how deeply nested arrays/objects can be before cJSON rejects to parse them.
 * This is to prevent stack overflows. */
#ifndef CJSON_NESTING_LIMIT
#define CJSON_NESTING_LIMIT 1000
#endif

/* returns the version of cJSON as a string */
CJSON_PUBLIC(const char*) cJSON_Version(void);

/* Supply malloc, realloc and free functions to cJSON */
CJSON_PUBLIC(void) cJSON_InitHooks(cJSON_Hooks* hooks);

/* Memory Management: the caller is always responsible to free the results from all variants of cJSON_Parse (with cJSON_Delete) and cJSON_Print (with stdlib free, cJSON_Hooks.free_fn, or cJSON_free as appropriate). The exception is cJSON_PrintPreallocated, where the caller has full responsibility of the buffer. */
/* Supply a block of JSON, and this returns a cJSON object you can interrogate. */
CJSON_PUBLIC(cJSON *) cJSON_Parse(const char *value);
/* ParseWithOpts allows you to require (and check) that the JSON is null terminated, and to retrieve the pointer to the final byte parsed. */
/* If you supply a ptr in return_parse_end and parsing fails, then return_parse_end will contain a pointer to the error so will match cJSON_GetErrorPtr(). */
CJSON_PUBLIC(cJSON *) cJSON_ParseWithOpts(const char *value, const char **return_parse_end, cJSON_bool require_null_terminated);

/* Render a cJSON entity to text for transfer/storage. */
CJSON_PUBLIC(char *) cJSON_Print(const cJSON *item);
/* Render a cJSON entity to text for transfer/storage without any formatting. */
CJSON_PUBLIC(char *) cJSON_PrintUnformatted(const cJSON *item);
/* Render a cJSON entity to text using a buffered strategy. prebuffer is a guess at the final size. guessing well reduces reallocation. fmt=0 gives unformatted, =1 gives formatted */
CJSON_PUBLIC(char *) cJSON_PrintBuffered(const cJSON *item, int prebuffer, cJSON_bool fmt);
/* Render a cJSON entity to text using a buffer already allocated in memory with given length. Returns 1 on success and 0 on failure. */
/* NOTE: cJSON is not always 100% accurate in estimating how much memory it will use, so to be safe allocate 5 bytes more than you actually need */
CJSON_PUBLIC(cJSON_bool) cJSON_PrintPreallocated(cJSON *item, char *buffer, const int length, const cJSON_bool format);
/* Delete a cJSON entity and all subentities. */
CJSON_PUBLIC(void) cJSON_Delete(cJSON *c);

/* Returns the number of items in an array (or object). */
CJSON_PUBLIC(int) cJSON_GetArraySize(const cJSON *array);
/* Retrieve item number "index" from array "array". Returns NULL if unsuccessful. */
CJSON_PUBLIC(cJSON *) cJSON_GetArrayItem(const cJSON *array, int index);
/* Get item "string" from object. Case insensitive. */
CJSON_PUBLIC(cJSON *) cJSON_GetObjectItem(const cJSON * const object, const char * const string);
CJSON_PUBLIC(cJSON *) cJSON_GetObjectItemCaseSensitive(const cJSON * const object, const char * const string);
CJSON_PUBLIC(cJSON_bool) cJSON_HasObjectItem(const cJSON *object, const char *string);
/* For analysing failed parses. This returns a pointer to the parse error. You'll probably need to look a few chars back to make sense of it. Defined when cJSON_Parse() returns 0. 0 when cJSON_Parse() succeeds. */
CJSON_PUBLIC(const char *) cJSON_GetErrorPtr(void);

/* Check if the item is a string and return its valuestring */
CJSON_PUBLIC(char *) cJSON_GetStringValue(cJSON *item);

/* These functions check the type of an item */
CJSON_PUBLIC(cJSON_bool) cJSON_IsInvalid(const cJSON * const item);
CJSON_PUBLIC(cJSON_bool) cJSON_IsFalse(const cJSON * const item);
CJSON_PUBLIC(cJSON_bool) cJSON_IsTrue(const cJSON * const item);
CJSON_PUBLIC(cJSON_bool) cJSON_IsBool(const cJSON * const item);
CJSON_PUBLIC(cJSON_bool) cJSON_IsNull(const cJSON * const item);
CJSON_PUBLIC(cJSON_bool) cJSON_IsNumber(const cJSON * const item);
CJSON_PUBLIC(cJSON_bool) cJSON_IsString(const cJSON * const item);
CJSON_PUBLIC(cJSON_bool) cJSON_IsArray(const cJSON * const item);
CJSON_PUBLIC(cJSON_bool) cJSON_IsObject(const cJSON * const item);
CJSON_PUBLIC(cJSON_bool) cJSON_IsRaw(const cJSON * const item);

/* These calls create a cJSON item of the appropriate type. */
CJSON_PUBLIC(cJSON *) cJSON_CreateNull(void);
CJSON_PUBLIC(cJSON *) cJSON_CreateTrue(void);
CJSON_PUBLIC(cJSON *) cJSON_CreateFalse(void);
CJSON_PUBLIC(cJSON *) cJSON_CreateBool(cJSON_bool boolean);
CJSON_PUBLIC(cJSON *) cJSON_CreateNumber(double num);
CJSON_PUBLIC(cJSON *) cJSON_CreateString(const char *string);
/* raw json */
CJSON_PUBLIC(cJSON *) cJSON_CreateRaw(const char *raw);
CJSON_PUBLIC(cJSON *) cJSON_CreateArray(void);
CJSON_PUBLIC(cJSON *) cJSON_CreateObject(void);

/* Create a string where valuestring references a string so
 * it will not be freed by cJSON_Delete */
CJSON_PUBLIC(cJSON *) cJSON_CreateStringReference(const char *string);
/* Create an object/arrray that only references it's elements so
 * they will not be freed by cJSON_Delete */
CJSON_PUBLIC(cJSON *) cJSON_CreateObjectReference(const cJSON *child);
CJSON_PUBLIC(cJSON *) cJSON_CreateArrayReference(const cJSON *child);

/* These utilities create an Array of count items. */
CJSON_PUBLIC(cJSON *) cJSON_CreateIntArray(const int *numbers, int count);
CJSON_PUBLIC(cJSON *) cJSON_CreateFloatArray(const float *numbers, int count);
CJSON_PUBLIC(cJSON *) cJSON_CreateDoubleArray(const double *numbers, int count);
CJSON_PUBLIC(cJSON *) cJSON_CreateStringArray(const char **strings, int count);

/* Append item to the specified array/object. */
CJSON_PUBLIC(void) cJSON_AddItemToArray(cJSON *array, cJSON *item);
CJSON_PUBLIC(void) cJSON_AddItemToObject(cJSON *object, const char *string, cJSON *item);
/* Use this when string is definitely const (i.e. a literal, or as good as), and will definitely survive the cJSON object.
 * WARNING: When this function was used, make sure to always check that (item->type & cJSON_StringIsConst) is zero before
 * writing to `item->string` */
CJSON_PUBLIC(void) cJSON_AddItemToObjectCS(cJSON *object, const char *string, cJSON *item);
/* Append reference to item to the specified array/object. Use this when you want to add an existing cJSON to a new cJSON, but don't want to corrupt your existing cJSON. */
CJSON_PUBLIC(void) cJSON_AddItemReferenceToArray(cJSON *array, cJSON *item);
CJSON_PUBLIC(void) cJSON_AddItemReferenceToObject(cJSON *object, const char *string, cJSON *item);

/* Remove/Detatch items from Arrays/Objects. */
CJSON_PUBLIC(cJSON *) cJSON_DetachItemViaPointer(cJSON *parent, cJSON * const item);
CJSON_PUBLIC(cJSON *) cJSON_DetachItemFromArray(cJSON *array, int which);
CJSON_PUBLIC(void) cJSON_DeleteItemFromArray(cJSON *array, int which);
CJSON_PUBLIC(cJSON *) cJSON_DetachItemFromObject(cJSON *object, const char *string);
CJSON_PUBLIC(cJSON *) cJSON_DetachItemFromObjectCaseSensitive(cJSON *object, const char *string);
CJSON_PUBLIC(void) cJSON_DeleteItemFromObject(cJSON *object, const char *string);
CJSON_PUBLIC(void) cJSON_DeleteItemFromObjectCaseSensitive(cJSON *object, const char *string);

/* Update array items. */
CJSON_PUBLIC(void) cJSON_InsertItemInArray(cJSON *array, int which, cJSON *newitem); /* Shifts pre-existing items to the right. */
CJSON_PUBLIC(cJSON_bool) cJSON_ReplaceItemViaPointer(cJSON * const parent, cJSON * const item, cJSON * replacement);
CJSON_PUBLIC(void) cJSON_ReplaceItemInArray(cJSON *array, int which, cJSON *newitem);
CJSON_PUBLIC(void) cJSON_ReplaceItemInObject(cJSON *object,const char *string,cJSON *newitem);
CJSON_PUBLIC(void) cJSON_ReplaceItemInObjectCaseSensitive(cJSON *object,const char *string,cJSON *newitem);

/* Duplicate a cJSON item */
CJSON_PUBLIC(cJSON *) cJSON_Duplicate(const cJSON *item, cJSON_bool recurse);
/* Duplicate will create a new, identical cJSON item to the one you pass, in new memory that will
need to be released. With recurse!=0, it will duplicate any children connected to the item.
The item->next and ->prev pointers are always zero on return from Duplicate. */
/* Recursively compare two cJSON items for equality. If either a or b is NULL or invalid, they will be considered unequal.
 * case_sensitive determines if object keys are treated case sensitive (1) or case insensitive (0) */
CJSON_PUBLIC(cJSON_bool) cJSON_Compare(const cJSON * const a, const cJSON * const b, const cJSON_bool case_sensitive);


CJSON_PUBLIC(void) cJSON_Minify(char *json);

/* Helper functions for creating and adding items to an object at the same time.
 * They return the added item or NULL on failure. */
CJSON_PUBLIC(cJSON*) cJSON_AddNullToObject(cJSON * const object, const char * const name);
CJSON_PUBLIC(cJSON*) cJSON_AddTrueToObject(cJSON * const object, const char * const name);
CJSON_PUBLIC(cJSON*) cJSON_AddFalseToObject(cJSON * const object, const char * const name);
CJSON_PUBLIC(cJSON*) cJSON_AddBoolToObject(cJSON * const object, const char * const name, const cJSON_bool boolean);
CJSON_PUBLIC(cJSON*) cJSON_AddNumberToObject(cJSON * const object, const char * const name, const double number);
CJSON_PUBLIC(cJSON*) cJSON_AddStringToObject(cJSON * const object, const char * const name, const char * const string);
CJSON_PUBLIC(cJSON*) cJSON_AddRawToObject(cJSON * const object, const char * const name, const char * const raw);
CJSON_PUBLIC(cJSON*) cJSON_AddObjectToObject(cJSON * const object, const char * const name);
CJSON_PUBLIC(cJSON*) cJSON_AddArrayToObject(cJSON * const object, const char * const name);

/* When assigning an integer value, it needs to be propagated to valuedouble too. */
#define cJSON_SetIntValue(object, number) ((object) ? (object)->valueint = (object)->valuedouble = (number) : (number))
/* helper for the cJSON_SetNumberValue macro */
CJSON_PUBLIC(double) cJSON_SetNumberHelper(cJSON *object, double number);
#define cJSON_SetNumberValue(object, number) ((object != NULL) ? cJSON_SetNumberHelper(object, (double)number) : (number))

/* Macro for iterating over an array or object */
#define cJSON_ArrayForEach(element, array) for(element = (array != NULL) ? (array)->child : NULL; element != NULL; element = element->next)

/* malloc/free objects using the malloc/free functions that have been set with cJSON_InitHooks */
CJSON_PUBLIC(void *) cJSON_malloc(size_t size);
CJSON_PUBLIC(void) cJSON_free(void *object);

#ifdef __cplusplus
}
#endif

#endif



================================================
FILE: src/CMakeLists.txt
================================================
# -*- cmake -*-
cmake_minimum_required(VERSION 3.5)
project("Useful Sqlite3 Extensions" LANGUAGES C VERSION 0.3)

set(CMAKE_MODULE_PATH ${PROJECT_SOURCE_DIR}/cmake)

include(CheckSymbolExists)

set(CMAKE_THREAD_PREFER_PTHREAD 1)
find_package(Threads)
find_package(ZLIB)

set(SAVED_DEFINITIONS ${CMAKE_REQUIRED_DEFINITIONS})
set(CMAKE_REQUIRED_DEFINITIONS -D_XOPEN_SOURCE)
check_symbol_exists(strptime "time.h" HAVE_STRPTIME)
set(CMAKE_REQUIRED_DEFINITIONS ${SAVED_DEFINITIONS})
find_package(ICU COMPONENTS data uc i18n)
if(ICU_FOUND)
  add_library(string_funcs MODULE icu_extras.c stripaccents.c egc_str_funcs.c
    more_str_funcs.c)
  set_target_properties(string_funcs PROPERTIES C_STANDARD 99
    C_STANDARD_REQUIRED ON)
  target_compile_options(string_funcs PRIVATE -Wall -Wextra)
  target_include_directories(string_funcs PRIVATE ${ICU_INCLUDE_DIRS}
    ${PROJECT_BINARY_DIR})
  target_link_libraries(string_funcs ${ICU_LIBRARIES})
endif()

find_package(OpenSSL)
if(OpenSSL_FOUND)
  set(SAVED_LIBRARIES ${CMAKE_REQUIRED_LIBRARIES})
  set(CMAKE_REQUIRED_LIBRARIES OpenSSL::Crypto)
  check_symbol_exists(EVP_MD_CTX_new "openssl/evp.h" HAVE_EVP_MD_CTX_NEW)
  set(CMAKE_REQUIRED_LIBRARIES ${SAVED_LIBRARIES})
  add_library(blob_funcs MODULE blob_funcs.c)
  set_target_properties(blob_funcs PROPERTIES C_STANDARD 99
    C_STANDARD_REQUIRED ON)
  target_compile_options(blob_funcs PRIVATE -Wall -Wextra)  
  target_include_directories(blob_funcs PRIVATE OpenSSL::Crypto ZLIB::ZLIB
    ${PROJECT_BINARY_DIR})
  target_link_libraries(blob_funcs OpenSSL::Crypto ZLIB::ZLIB Threads::Threads)
endif()

find_package(PCRE2)
if(Pcre2_8_FOUND OR Pcre2_8_FOUND)
  add_library(pcre2_funcs MODULE pcre2_funcs.c)
  set_target_properties(pcre2_funcs PROPERTIES C_STANDARD 99
    C_STANDARD_REQUIRED ON)
  if(Pcre2_8_FOUND)
    set(HAVE_PCRE2_8 1)
    target_link_libraries(pcre2_funcs ${PCRE2_8})
  endif()
  if(Pcre2_16_FOUND)
    set(HAVE_PCRE1_16 1)
    target_link_libraries(pcre2_funcs ${PCRE2_16})
  endif()
  target_include_directories(pcre2_funcs PRIVATE ${PCRE2_INCLUDE_DIR}
    ${PROJECT_BINARY_DIR})
endif()

check_symbol_exists(regcomp "regex.h" HAVE_REGCOMP)
if(HAVE_REGCOMP)
  add_library(posix_re_funcs MODULE posix_re_funcs.c)
  set_target_properties(posix_re_funcs PROPERTIES C_STANDARD 99
    C_STANDARD_REQUIRED ON)
  target_include_directories(posix_re_funcs PRIVATE "${PROJECT_BINARY_DIR}")
endif()

add_library(math_funcs MODULE math_funcs.c)
set_target_properties(math_funcs PROPERTIES C_STANDARD 99
  C_STANDARD_REQUIRED ON)
target_compile_options(math_funcs PRIVATE -Wall -Wextra)

add_library(bloom_filter1 MODULE bloom_filter.c)
set_target_properties(bloom_filter1 PROPERTIES C_STANDARD 99
  C_STANDARD_REQUIRED ON)
target_compile_options(bloom_filter1 PRIVATE -Wall -Wextra
  -Wno-implicit-fallthrough)

add_library(json_funcs MODULE json_funcs.c cJSON.c)
set_target_properties(json_funcs PROPERTIES C_STANDARD 99
  C_STANDARD_REQUIRED ON)
target_compile_options(json_funcs PRIVATE -Wall -Wextra)

configure_file(config.h.in config.h)



================================================
FILE: src/config.h.in
================================================
/* Config file for compile-time options */
#pragma once

#cmakedefine HAVE_PCRE2_8
#cmakedefine HAVE_PCRE2_16
#cmakedefine CMAKE_USE_PTHREADS_INIT
#cmakedefine ZLIB_FOUND
#cmakedefine HAVE_REGCOMP
#cmakedefine HAVE_STRPTIME
#cmakedefine HAVE_EVP_MD_CTX_NEW



================================================
FILE: src/egc_str_funcs.c
================================================
/*
Copyright 2018-2019 Shawn Wagner

Permission is hereby granted, free of charge, to any person obtaining
a copy of this software and associated documentation files (the
"Software"), to deal in the Software without restriction, including
without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject to
the following conditions:

The above copyright notice and this permission notice shall be
included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/

/* String functions that deal with extended grapheme clusters instead
   of strings as a whole. */

#include <assert.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

#include <unicode/ubrk.h>
#include <unicode/ustring.h>
#include <unicode/utypes.h>

#include <sqlite3ext.h>
SQLITE_EXTENSION_INIT3

void icuFunctionError(
    sqlite3_context *pCtx, /* SQLite scalar function context */
    const char *zName,     /* Name of ICU function that failed */
    UErrorCode e           /* Error code returned by ICU function */
);

static void closeBreakIterator(void *v) {
  UBreakIterator *bi = v;
  ubrk_close(bi);
}

UBreakIterator *default_charbreak = NULL;

static void sf_gclength16(sqlite3_context *c, int nArg __attribute__((unused)),
                          sqlite3_value **apArg) {
  assert(nArg == 1 || nArg == 2);

  if (sqlite3_value_type(apArg[0]) == SQLITE_NULL) {
    return;
  }
  if (nArg == 2 && sqlite3_value_type(apArg[1]) == SQLITE_NULL) {
    return;
  }

  const UChar *utf16 = sqlite3_value_text16(apArg[0]);
  if (!utf16) {
    return;
  }
  UBreakIterator *bi = NULL;
  UErrorCode status = U_ZERO_ERROR;
  if (nArg == 2) {
    bi = sqlite3_get_auxdata(c, 1);
    if (!bi) {
      const char *locale = (const char *)sqlite3_value_text(apArg[1]);
      if (!locale) {
        return;
      }
      bi = ubrk_open(UBRK_CHARACTER, locale, NULL, 0, &status);
      if (U_FAILURE(status)) {
        icuFunctionError(c, "ubrk_open", status);
        return;
      }
      sqlite3_set_auxdata(c, 1, bi, closeBreakIterator);
    }
  } else {
    bi = ubrk_safeClone(default_charbreak, NULL, NULL, &status);
    if (U_FAILURE(status)) {
      icuFunctionError(c, "ubrk_safeClone", status);
      return;
    }
  }

  ubrk_setText(bi, utf16, -1, &status);
  if (U_FAILURE(status)) {
    icuFunctionError(c, "ubrk_setText", status);
    return;
  }

  int len = 0;
  while (ubrk_next(bi) != UBRK_DONE) {
    len += 1;
  }

  if (nArg == 1) {
    ubrk_close(bi);
  }
  sqlite3_result_int(c, len);
}

static void sf_gcleft16(sqlite3_context *c, int nArg __attribute__((unused)),
                        sqlite3_value **apArg) {
  assert(nArg == 2 || nArg == 3);

  if (sqlite3_value_type(apArg[0]) == SQLITE_NULL ||
      sqlite3_value_type(apArg[1]) == SQLITE_NULL) {
    return;
  }
  if (nArg == 3 && sqlite3_value_type(apArg[2]) == SQLITE_NULL) {
    return;
  }

  int n = sqlite3_value_int(apArg[1]);
  if (n == 0) {
    sqlite3_result_text(c, "", 0, SQLITE_STATIC);
    return;
  }

  const UChar *utf16 = sqlite3_value_text16(apArg[0]);
  if (!utf16) {
    return;
  }

  UErrorCode status = U_ZERO_ERROR;
  UBreakIterator *bi;
  if (nArg == 3) {
    bi = sqlite3_get_auxdata(c, 2);
    if (!bi) {
      const char *locale = (const char *)sqlite3_value_text(apArg[2]);
      if (!locale) {
        return;
      }
      bi = ubrk_open(UBRK_CHARACTER, locale, NULL, 0, &status);
      if (U_FAILURE(status)) {
        icuFunctionError(c, "ubrk_open", status);
        return;
      }
      sqlite3_set_auxdata(c, 2, bi, closeBreakIterator);
    }
  } else {
    bi = ubrk_safeClone(default_charbreak, NULL, NULL, &status);
    if (U_FAILURE(status)) {
      icuFunctionError(c, "ubrk_safeClone", status);
      return;
    }
  }

  ubrk_setText(bi, utf16, -1, &status);
  if (U_FAILURE(status)) {
    icuFunctionError(c, "ubrk_setText", status);
    return;
  }

  int32_t endlen = 0;
  if (n > 0) {
    int32_t off;
    while (n > 0 && (off = ubrk_next(bi)) != UBRK_DONE) {
      endlen = off;
      n -= 1;
    }
  } else if (n < 0) {
    int32_t off;
    ubrk_last(bi);
    while (n < 0 && (off = ubrk_previous(bi)) != UBRK_DONE) {
      endlen = off;
      n += 1;
    }
  }
  sqlite3_result_text16(c, utf16, endlen * 2, SQLITE_TRANSIENT);
  if (nArg == 2) {
    ubrk_close(bi);
  }
}

static void sf_gcright16(sqlite3_context *c, int nArg __attribute__((unused)),
                         sqlite3_value **apArg) {

  assert(nArg == 2 || nArg == 3);

  if (sqlite3_value_type(apArg[0]) == SQLITE_NULL ||
      sqlite3_value_type(apArg[1]) == SQLITE_NULL) {
    return;
  }

  int n = sqlite3_value_int(apArg[1]);
  if (n == 0) {
    sqlite3_result_text(c, "", 0, SQLITE_STATIC);
    return;
  }

  const UChar *utf16 = sqlite3_value_text16(apArg[0]);
  if (!utf16) {
    return;
  }

  UErrorCode status = U_ZERO_ERROR;
  UBreakIterator *bi;
  if (nArg == 3) {
    bi = sqlite3_get_auxdata(c, 2);
    if (!bi) {
      const char *locale = (const char *)sqlite3_value_text(apArg[2]);
      if (!locale) {
        return;
      }
      bi = ubrk_open(UBRK_CHARACTER, locale, NULL, 0, &status);
      if (U_FAILURE(status)) {
        icuFunctionError(c, "ubrk_open", status);
        return;
      }
      sqlite3_set_auxdata(c, 2, bi, closeBreakIterator);
    }
  } else {
    bi = ubrk_safeClone(default_charbreak, NULL, NULL, &status);
    if (U_FAILURE(status)) {
      icuFunctionError(c, "ubrk_safeClone", status);
      return;
    }
  }

  ubrk_setText(bi, utf16, -1, &status);
  if (U_FAILURE(status)) {
    icuFunctionError(c, "ubrk_setText", status);
    return;
  }

  int32_t off = 0, nextoff = 0;
  if (n > 0) {
    ubrk_last(bi);
    while (n > 0 && (nextoff = ubrk_previous(bi)) != UBRK_DONE) {
      n -= 1;
      off = nextoff;
    }
  } else {
    while (n < 0 && (nextoff = ubrk_next(bi)) != UBRK_DONE) {
      off = nextoff;
      n += 1;
    }
  }
  sqlite3_result_text16(c, utf16 + off, -1, SQLITE_TRANSIENT);
  if (nArg == 2) {
    ubrk_close(bi);
  }
}

static void sf_gcsubstr16(sqlite3_context *c, int nArg, sqlite3_value **apArg) {
  assert(nArg >= 2 && nArg <= 4);
  if (sqlite3_value_type(apArg[0]) == SQLITE_NULL ||
      sqlite3_value_type(apArg[1]) == SQLITE_NULL) {
    return;
  }
  if (nArg >= 3 && sqlite3_value_type(apArg[2]) == SQLITE_NULL) {
    return;
  }
  if (nArg == 4 && sqlite3_value_type(apArg[3]) == SQLITE_NULL) {
    return;
  }

  const UChar *utf16 = sqlite3_value_text16(apArg[0]);
  if (!utf16) {
    return;
  }

  int start_pos = sqlite3_value_int(apArg[1]);
  if (start_pos <= 0) {
    sqlite3_result_error_code(c, SQLITE_RANGE);
    return;
  }
  start_pos -= 1;

  int sublen = -1;
  if (nArg == 3) {
    sublen = sqlite3_value_int(apArg[2]);
    if (sublen < -1) {
      sqlite3_result_error_code(c, SQLITE_RANGE);
      return;
    }
  }

  UErrorCode status = U_ZERO_ERROR;
  UBreakIterator *bi;
  if (nArg == 4) {
    bi = sqlite3_get_auxdata(c, 3);
    if (!bi) {
      const char *locale = (const char *)sqlite3_value_text(apArg[3]);
      if (!locale) {
        return;
      }
      bi = ubrk_open(UBRK_CHARACTER, locale, NULL, 0, &status);
      if (U_FAILURE(status)) {
        icuFunctionError(c, "ubrk_open", status);
        return;
      }
      sqlite3_set_auxdata(c, 3, bi, closeBreakIterator);
    }
  } else {
    bi = ubrk_safeClone(default_charbreak, NULL, NULL, &status);
    if (U_FAILURE(status)) {
      icuFunctionError(c, "ubrk_safeClone", status);
      return;
    }
  }

  ubrk_setText(bi, utf16, -1, &status);
  if (U_FAILURE(status)) {
    icuFunctionError(c, "ubrk_setText", status);
    return;
  }

  int32_t off = 0, nextoff = 0;
  while (start_pos > 0 && (nextoff = ubrk_next(bi)) != UBRK_DONE) {
    start_pos -= 1;
    off = nextoff;
  }

  if (sublen == -1) {
    sqlite3_result_text16(c, utf16 + off, -1, SQLITE_TRANSIENT);
    if (nArg != 4) {
      ubrk_close(bi);
    }
    return;
  }

  int32_t endoff = 0;
  while (sublen > 0 && (nextoff = ubrk_next(bi)) != UBRK_DONE) {
    sublen -= 1;
    endoff = nextoff;
  }

  sqlite3_result_text16(c, utf16 + off, (endoff - off) * 2, SQLITE_TRANSIENT);
  if (nArg != 4) {
    ubrk_close(bi);
  }
}

/* Eponymous-only virtual tables to break strings */

struct break_vtab {
  sqlite3_vtab base;
  UBreakIteratorType type;
};

struct break_cursor {
  sqlite3_vtab_cursor base;
  sqlite3_int64 rowid;
  char *locale;
  UChar *utf16;
  UBreakIterator *bi;
  int cps_seen;
  int32_t start_off;
  int32_t end_off;
};

static int breakConnect(sqlite3 *db, void *pAux,
                        int argc __attribute__((unused)),
                        const char *const *argv __attribute__((unused)),
                        sqlite3_vtab **ppVtab,
                        char **pzErr __attribute__((unused))) {
  struct break_vtab *bv;
  int rc;

#define BREAK_COLUMN_VALUE 0
#define BREAK_COLUMN_START 1
#define BREAK_COLUMN_LEN 2
#define BREAK_COLUMN_TXT 3
#define BREAK_COLUMN_LOCALE 4

  rc = sqlite3_declare_vtab(db, "CREATE TABLE x(value TEXT, start INTEGER, "
                                "len INTEGER, txt hidden, locale hidden)");
  if (rc != SQLITE_OK) {
    return rc;
  }
  bv = sqlite3_malloc(sizeof *bv);
  if (!bv) {
    return SQLITE_NOMEM;
  }
  *ppVtab = &bv->base;
  memset(bv, 0, sizeof *bv);
  bv->type = (UBreakIteratorType)pAux;
  return SQLITE_OK;
}

static int breakDisconnect(sqlite3_vtab *pVtab) {
  sqlite3_free(pVtab);
  return SQLITE_OK;
}

static int breakOpen(sqlite3_vtab *p __attribute__((unused)),
                     sqlite3_vtab_cursor **ppCursor) {
  struct break_cursor *bc = sqlite3_malloc(sizeof *bc);
  if (!bc) {
    return SQLITE_NOMEM;
  }
  memset(bc, 0, sizeof *bc);
  *ppCursor = &bc->base;
  bc->cps_seen = -1;
  return SQLITE_OK;
}

static int breakClose(sqlite3_vtab_cursor *cur) {
  struct break_cursor *bc = (struct break_cursor *)cur;
  if (bc->bi) {
    ubrk_close(bc->bi);
  }
  sqlite3_free(bc->utf16);
  sqlite3_free(bc->locale);
  sqlite3_free(bc);
  return SQLITE_OK;
}

static int breakNext(sqlite3_vtab_cursor *cur) {
  struct break_cursor *bc = (struct break_cursor *)cur;
  if (bc->cps_seen >= 0) {
    bc->cps_seen +=
        u_countChar32(bc->utf16 + bc->start_off, bc->end_off - bc->start_off);
  }
  bc->start_off = bc->end_off;
  bc->end_off = ubrk_next(bc->bi);
  bc->rowid += 1;
  return SQLITE_OK;
}

static int breakColumn(sqlite3_vtab_cursor *cur, sqlite3_context *c, int i) {
  struct break_cursor *bc = (struct break_cursor *)cur;
  switch (i) {
  default:
  case BREAK_COLUMN_VALUE:
    if (bc->utf16) {
      sqlite3_result_text16(c, bc->utf16 + bc->start_off,
                            (bc->end_off - bc->start_off) * 2, SQLITE_STATIC);
    }
    break;
  case BREAK_COLUMN_START:
    if (bc->utf16) {
      if (bc->cps_seen < 0) {
        bc->cps_seen = u_countChar32(bc->utf16, bc->start_off);
      }
      sqlite3_result_int(c, bc->cps_seen + 1);
    }
    break;
  case BREAK_COLUMN_LEN:
    if (bc->utf16) {
      sqlite3_result_int(c, u_countChar32(bc->utf16 + bc->start_off,
                                          bc->end_off - bc->start_off));
    }
    break;
  case BREAK_COLUMN_TXT:
    if (bc->utf16) {
      sqlite3_result_text16(c, bc->utf16, -1, SQLITE_STATIC);
    }
    break;
  case BREAK_COLUMN_LOCALE:
    if (bc->locale) {
      sqlite3_result_text(c, bc->locale, -1, SQLITE_STATIC);
    }
    break;
  }
  return SQLITE_OK;
}

static int breakRowid(sqlite3_vtab_cursor *cur, sqlite3_int64 *pRowid) {
  struct break_cursor *bc = (struct break_cursor *)cur;
  *pRowid = bc->rowid;
  return SQLITE_OK;
}

static int breakEof(sqlite3_vtab_cursor *cur) {
  struct break_cursor *bc = (struct break_cursor *)cur;
  return !bc->utf16 || bc->end_off == UBRK_DONE;
}

static int breakFilter(sqlite3_vtab_cursor *cur,
                       int idxNum __attribute__((unused)),
                       const char *idxStr __attribute__((unused)), int argc,
                       sqlite3_value **argv) {
  struct break_cursor *bc = (struct break_cursor *)cur;
  struct break_vtab *bv = (struct break_vtab *)bc->base.pVtab;

  assert(argc == 1 || argc == 2);

  if (sqlite3_value_type(argv[0]) == SQLITE_NULL) {
    return SQLITE_OK;
  }

  if (argc == 2) {
    if (sqlite3_value_type(argv[1]) != SQLITE_NULL) {
      bc->locale = sqlite3_mprintf("%s", sqlite3_value_text(argv[1]));
      if (!bc->locale) {
        return SQLITE_NOMEM;
      }
    }
  }

  const void *u16 = sqlite3_value_text16(argv[0]);
  if (!u16) {
    return SQLITE_OK;
  }
  int32_t len = sqlite3_value_bytes16(argv[0]);
  bc->utf16 = sqlite3_malloc(len + 2);
  memcpy(bc->utf16, u16, len);
  bc->utf16[len / 2] = 0;

  UErrorCode status = U_ZERO_ERROR;
  bc->bi = ubrk_open(bv->type, bc->locale, bc->utf16, -1, &status);
  if (U_FAILURE(status)) {
    sqlite3_free(bc->utf16);
    sqlite3_free(bc->locale);
    sqlite3_free(cur->pVtab->zErrMsg);
    cur->pVtab->zErrMsg =
        sqlite3_mprintf("ICU error: ubrk_open(): %s", u_errorName(status));
    return SQLITE_ERROR;
  }

  return breakNext(cur);
}

static int breakBestIndex(sqlite3_vtab *tab, sqlite3_index_info *pIdxInfo) {
  int aIdx[2] = {-1, -1};
  int unusableMask = 0;
  int idxMask = 0;
  _Bool start_requested = 0, len_requested = 0;
  const struct sqlite3_index_constraint *pConstraint = pIdxInfo->aConstraint;
  for (int i = 0; i < pIdxInfo->nConstraint; i++, pConstraint++) {
    if (pConstraint->iColumn < BREAK_COLUMN_TXT) {
      if (pConstraint->usable && pConstraint->iColumn == BREAK_COLUMN_START) {
        start_requested = 1;
      } else

          if (pConstraint->usable && pConstraint->iColumn == BREAK_COLUMN_LEN) {
        len_requested = 1;
      }
      continue;
    }
    int iCol = pConstraint->iColumn - BREAK_COLUMN_TXT;
    int iMask = 1 << iCol;
    if (pConstraint->usable == 0) {
      unusableMask |= iMask;
    } else if (pConstraint->op == SQLITE_INDEX_CONSTRAINT_EQ) {
      aIdx[iCol] = i;
      idxMask |= iMask;
    }
  }
  if ((unusableMask & !idxMask) != 0) {
    return SQLITE_CONSTRAINT;
  }
  if (aIdx[0] >= 0) {
    int i = aIdx[0];
    pIdxInfo->aConstraintUsage[i].argvIndex = 1;
    pIdxInfo->aConstraintUsage[i].omit = 1;
  }
  if (aIdx[1] >= 0) {
    int i = aIdx[1];
    pIdxInfo->aConstraintUsage[i].argvIndex = 2;
    pIdxInfo->aConstraintUsage[i].omit = 1;
    pIdxInfo->idxNum = 2;
  } else {
    pIdxInfo->idxNum = 1;
  }

#undef MAX
#define MAX(a, b) (a) > (b) ? (a) : (b)
  int cost = 0;
  struct break_vtab *bv = (struct break_vtab *)tab;
  if (bv->type == UBRK_CHARACTER) {
    cost = 500;
  } else {
    /* Word and sentence tokenizing is more expensive as there are
     * more code points to scan. */
    cost = 1200;
  }
  /* Counting the code points to the starting index is only done on
   * demand; expensive once for the first count if not done from the
   * very beginning. Amortize the costs. */
  if (start_requested) {
    if (bv->type == UBRK_CHARACTER) {
      cost += 100;
    } else if (bv->type == UBRK_WORD) {
      cost += 200;
    } else {
      cost += 400;
    }
  }
  /* Counting the code points in the token is more expensive for words
   * and sentences than characters. */
  if (len_requested) {
    if (bv->type == UBRK_CHARACTER) {
      cost = MAX(cost, 550);
    } else if (bv->type == UBRK_WORD) {
      cost = MAX(cost, 1500);
    } else {
      cost = MAX(cost, 4000);
    }
  }
  pIdxInfo->estimatedCost = cost;
  return SQLITE_OK;
}

static sqlite3_module breakModule = {
    1,               // iVersion
    0,               // xCreate
    breakConnect,    // xConnect
    breakBestIndex,  // xBestIndex
    breakDisconnect, // xDisconnect
    0,               // xDestroy
    breakOpen,       // xOpen
    breakClose,      // xClose
    breakFilter,     // xFilter
    breakNext,       // xNext
    breakEof,        // xEof
    breakColumn,     // xColumn
    breakRowid,      // xRowid
    0,               // xUpdate
    0,               // xBegin
    0,               // xSync
    0,               // xCommit
    0,               // xRollback
    0,               // xFindFunction
    0,               // xRename
    0,               // xSavepoint
    0,               // xRelease
    0,               // xRollbackTo
#if SQLITE_VERSION_NUMBER >= 3026000
    0, // xShadowName
#endif
};

int sf_egc_init(sqlite3 *db, char **pzErrMsg) {
  const struct Scalar {
    const char *zName;  /* Function name */
    int nArg;           /* Number of arguments */
    unsigned short enc; /* Optimal text encoding */
    void *iContext;     /* sqlite3_user_data() context */
    void (*xFunc)(sqlite3_context *, int, sqlite3_value **);
  } scalars[] = {
      {"gclength", 1, SQLITE_UTF16 | SQLITE_DETERMINISTIC, NULL, sf_gclength16},
      {"gclength", 2, SQLITE_UTF16 | SQLITE_DETERMINISTIC, NULL, sf_gclength16},
      {"gcleft", 2, SQLITE_UTF16 | SQLITE_DETERMINISTIC, NULL, sf_gcleft16},
      {"gcleft", 3, SQLITE_UTF16 | SQLITE_DETERMINISTIC, NULL, sf_gcleft16},
      {"gcright", 2, SQLITE_UTF16 | SQLITE_DETERMINISTIC, NULL, sf_gcright16},
      {"gcright", 3, SQLITE_UTF16 | SQLITE_DETERMINISTIC, NULL, sf_gcright16},
      {"gcsubstr", 2, SQLITE_UTF16 | SQLITE_DETERMINISTIC, NULL, sf_gcsubstr16},
      {"gcsubstr", 3, SQLITE_UTF16 | SQLITE_DETERMINISTIC, NULL, sf_gcsubstr16},
      {"gcsubstr", 4, SQLITE_UTF16 | SQLITE_DETERMINISTIC, NULL, sf_gcsubstr16},
      {NULL, -1, 0, NULL, NULL}};
  const struct Break {
    const char *name;
    UBreakIteratorType type;
  } breaks[] = {{"graphemes", UBRK_CHARACTER},
                {"words", UBRK_WORD},
                {"sentences", UBRK_SENTENCE},
                {"lines", UBRK_LINE},
                {NULL, UBRK_COUNT}};
  int rc = SQLITE_OK;
  UErrorCode status = U_ZERO_ERROR;

  if (!default_charbreak) {
    default_charbreak = ubrk_open(UBRK_CHARACTER, NULL, NULL, 0, &status);
    if (U_FAILURE(status)) {
      if (pzErrMsg) {
        *pzErrMsg =
            sqlite3_mprintf("ICU error: ubrk_open(): %s", u_errorName(status));
      }
      return SQLITE_ERROR;
    }
  }

  for (int i = 0; rc == SQLITE_OK && scalars[i].zName; i += 1) {
    const struct Scalar *p = &scalars[i];
    rc = sqlite3_create_function(db, p->zName, p->nArg, p->enc, p->iContext,
                                 p->xFunc, 0, 0);
  }

  for (int i = 0; rc == SQLITE_OK && breaks[i].name; i += 1) {
    rc = sqlite3_create_module(db, breaks[i].name, &breakModule,
                               (void *)breaks[i].type);
  }

  return rc;
}



================================================
FILE: src/group.c
================================================
/*
Copyright 2018 Shawn Wagner

Permission is hereby granted, free of charge, to any person obtaining
a copy of this software and associated documentation files (the
"Software"), to deal in the Software without restriction, including
without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject to
the following conditions:

The above copyright notice and this permission notice shall be
included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/

#define _GNU_SOURCE

#include <errno.h>
#include <grp.h>
#include <stdlib.h>
#include <string.h>
#include <sys/types.h>
#include <unistd.h>

#include <sqlite3ext.h>
SQLITE_EXTENSION_INIT3

#include "nss_tables.h"

/* /etc/group table */

struct group_vtab {
  sqlite3_vtab vtab;
};

struct group_cursor {
  sqlite3_vtab *vtab;
  struct group *gr;
  struct group grstorage;
  char *buf;
  int bufsize;
  int memidx;
  sqlite3_int64 rowid;
  _Bool specific;
};

static int group_connect(sqlite3 *db, void *pAux __attribute__((unused)),
                         int argc __attribute__((unused)),
                         const char *const *argv __attribute__((unused)),
                         sqlite3_vtab **ppVTab, char **pzErr) {
  int status;
  status = sqlite3_declare_vtab(
      db, "CREATE TABLE etc_group(name TEXT NOT NULL, password TEXT, "
          "gid INTEGER NOT NULL, member TEXT)");
  if (status != SQLITE_OK) {
    *pzErr = sqlite3_mprintf("%s", sqlite3_errstr(status));
    return SQLITE_ERROR;
  }

  *ppVTab = sqlite3_malloc(sizeof(struct group_vtab));
  (*ppVTab)->pModule = &group_funcs;
  (*ppVTab)->nRef = 0;
  (*ppVTab)->zErrMsg = NULL;
  return SQLITE_OK;
}

static int group_bestindex(sqlite3_vtab *tab __attribute__((unused)),
                           sqlite3_index_info *info) {
  info->idxNum = 0;
  for (int n = 0; n < info->nConstraint; n += 1) {
    if (info->aConstraint[n].usable == 0) {
      continue;
    }
    if (info->aConstraint[n].iColumn == -1 ||
        info->aConstraint[n].iColumn == 2) {
      if (info->aConstraint[n].op == SQLITE_INDEX_CONSTRAINT_EQ) {
        info->idxNum |= 1;
        info->aConstraintUsage[n].argvIndex = 1;
        info->aConstraintUsage[n].omit = 1;
      }
    } else if (info->aConstraint[n].iColumn == 0) {
      if (info->aConstraint[n].op == SQLITE_INDEX_CONSTRAINT_EQ) {
        info->idxNum |= 2;
        info->aConstraintUsage[n].argvIndex = 1;
        info->aConstraintUsage[n].omit = 1;
      }
    }
  }
  if (info->idxNum) {
    info->estimatedCost = 10;
    info->estimatedRows = 1;
    info->idxFlags = SQLITE_INDEX_SCAN_UNIQUE;
  } else {
    info->estimatedCost = 200;
  }
  return SQLITE_OK;
}

static int group_disconnect(sqlite3_vtab *tab) {
  sqlite3_free(tab);
  return SQLITE_OK;
}

static int group_open(sqlite3_vtab *tab, sqlite3_vtab_cursor **curs) {
  struct group_cursor *c = sqlite3_malloc(sizeof(struct group_cursor));
  if (!c) {
    if (tab->zErrMsg) {
      sqlite3_free(tab->zErrMsg);
    }
    tab->zErrMsg = sqlite3_mprintf("Out of memory");
    return SQLITE_NOMEM;
  }
  c->vtab = tab;
  c->bufsize = 4096;
  c->buf = sqlite3_malloc(c->bufsize);
  if (!c->buf) {
    if (tab->zErrMsg) {
      sqlite3_free(tab->zErrMsg);
    }
    sqlite3_free(c);
    tab->zErrMsg = sqlite3_mprintf("Out of memory");
    return SQLITE_NOMEM;
  }
  c->rowid = 0;
  
  *curs = (sqlite3_vtab_cursor *)c;
  return SQLITE_OK;
}

static int group_close(sqlite3_vtab_cursor *vc) {
  struct group_cursor *c = (struct group_cursor *)vc;
  if (!c->specific) {
    endgrent();
  }
  sqlite3_free(c->buf);
  sqlite3_free(c);
  return SQLITE_OK;
}

int group_eof(sqlite3_vtab_cursor *vc) {
  struct group_cursor *c = (struct group_cursor *)vc;
  return c->gr == NULL;
}

static int group_filter(sqlite3_vtab_cursor *vc, int idxNum,
                        const char *idxStr __attribute__((unused)),
                        int argc __attribute__((unused)),
                        sqlite3_value **argv) {
  struct group_cursor *c = (struct group_cursor *)vc;

  c->memidx = 0;

  if (idxNum == 3) {
    if (c->vtab->zErrMsg) {
      sqlite3_free(c->vtab->zErrMsg);
    }
    c->vtab->zErrMsg = sqlite3_mprintf("cannot search for a gid AND groupname");
    return SQLITE_CONSTRAINT_VTAB;
  }

  if (idxNum == 1) {
    gid_t gid = sqlite3_value_int(argv[0]);
    c->specific = 1;
    while (1) {
      if (getgrgid_r(gid, &(c->grstorage), c->buf, c->bufsize, &(c->gr)) < 0) {
        if (errno == ERANGE) {
          c->buf = sqlite3_realloc(c->buf, c->bufsize * 2);
          c->bufsize *= 2;
        } else if (errno == ENOENT || errno == ESRCH) {
          break;
        } else {
          if (c->vtab->zErrMsg) {
            sqlite3_free(c->vtab->zErrMsg);
          }
          c->vtab->zErrMsg = sqlite3_mprintf("getgrgid_r: %s", strerror(errno));
          return SQLITE_ERROR;
        }
      } else {
        break;
      }
    }
    return SQLITE_OK;
  }

  if (idxNum == 2) {
    const char *username = (const char *)sqlite3_value_text(argv[0]);
    c->specific = 1;
    while (1) {
      if (getgrnam_r(username, &(c->grstorage), c->buf, c->bufsize, &(c->gr)) <
          0) {
        if (errno == ERANGE) {
          c->buf = sqlite3_realloc(c->buf, c->bufsize * 2);
          c->bufsize *= 2;
        } else if (errno == ENOENT || errno == ESRCH) {
          break;
        } else {
          if (c->vtab->zErrMsg) {
            sqlite3_free(c->vtab->zErrMsg);
          }
          c->vtab->zErrMsg = sqlite3_mprintf("getgrnam_r: %s", strerror(errno));
          return SQLITE_ERROR;
        }
      } else {
        break;
      }
    }
    return SQLITE_OK;
  }

  setgrent();
  c->specific = 0;
  while (1) {
    if (getgrent_r(&(c->grstorage), c->buf, c->bufsize, &(c->gr)) < 0) {
      if (errno == ERANGE) {
        c->buf = sqlite3_realloc(c->buf, c->bufsize * 2);
        c->bufsize *= 2;
      } else if (errno == ENOENT || errno == ESRCH) {
        break;
      } else {
        if (c->vtab->zErrMsg) {
          sqlite3_free(c->vtab->zErrMsg);
        }
        c->vtab->zErrMsg = sqlite3_mprintf("getgrent_r: %s", strerror(errno));
        return SQLITE_ERROR;
      }
    } else {
      break;
    }
  }
  return SQLITE_OK;
}

static int group_next(sqlite3_vtab_cursor *vc) {
  struct group_cursor *c = (struct group_cursor *)vc;

  // One row per member of the current group
  if (c->gr->gr_mem[c->memidx] != NULL) {
    c->memidx += 1;
    if (c->gr->gr_mem[c->memidx]) {
      c->rowid += 1;
      return SQLITE_OK;
    }
  }

  // EOF if done with members of an indexed group.
  if (c->specific) {
    c->gr = NULL;
    return SQLITE_OK;
  }

  // Get the next group.
  while (1) {
    if (getgrent_r(&(c->grstorage), c->buf, c->bufsize, &(c->gr)) < 0) {
      if (errno == ERANGE) {
        c->buf = sqlite3_realloc(c->buf, c->bufsize * 2);
        c->bufsize *= 2;
      } else {
        if (c->vtab->zErrMsg) {
          sqlite3_free(c->vtab->zErrMsg);
        }
        c->vtab->zErrMsg = sqlite3_mprintf("getgrent_r: %s", strerror(errno));
        return SQLITE_ERROR;
      }
    } else {
      break;
    }
  }
  c->memidx = 0;
  c->rowid += 1;
  return SQLITE_OK;
}

static int group_column(sqlite3_vtab_cursor *vc, sqlite3_context *p, int n) {
  struct group_cursor *c = (struct group_cursor *)vc;
  switch (n) {
  case 0:
    sqlite3_result_text(p, c->gr->gr_name, -1, SQLITE_TRANSIENT);
    break;
  case 1:
    if (c->gr->gr_passwd) {
      sqlite3_result_text(p, c->gr->gr_passwd, -1, SQLITE_TRANSIENT);
    } else {
      sqlite3_result_null(p);
    }
    break;
  case 2:
    sqlite3_result_int(p, c->gr->gr_gid);
    break;
  case 3:
    if (c->gr->gr_mem[c->memidx]) {
      sqlite3_result_text(p, c->gr->gr_mem[c->memidx], -1, SQLITE_TRANSIENT);
    } else {
      sqlite3_result_null(p);
    }
    break;
  default:
    if (c->vtab->zErrMsg) {
      sqlite3_free(c->vtab->zErrMsg);
    }
    c->vtab->zErrMsg = sqlite3_mprintf("Column out of range");
    return SQLITE_RANGE;
  }
  return SQLITE_OK;
}

static int group_rowid(sqlite3_vtab_cursor *vc, sqlite3_int64 *pRowId) {
  struct group_cursor *c = (struct group_cursor *)vc;
  *pRowId = c->rowid;
  return SQLITE_OK;
}

static int group_rename(sqlite3_vtab *tab __attribute__((unused)),
                        const char *newname __attribute__((unused))) {
  return SQLITE_OK;
}

struct sqlite3_module group_funcs = {1,
                                     group_connect,
                                     group_connect,
                                     group_bestindex,
                                     group_disconnect,
                                     group_disconnect,
                                     group_open,
                                     group_close,
                                     group_filter,
                                     group_next,
                                     group_eof,
                                     group_column,
                                     group_rowid,
                                     NULL,
                                     NULL,
                                     NULL,
                                     NULL,
                                     NULL,
                                     NULL,
                                     group_rename,
                                     NULL,
                                     NULL,
                                     NULL};



================================================
FILE: src/icu_extras.c
================================================
#include <stdio.h>
/*
Copyright 2018 Shawn Wagner

Permission is hereby granted, free of charge, to any person obtaining
a copy of this software and associated documentation files (the
"Software"), to deal in the Software without restriction, including
without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject to
the following conditions:

The above copyright notice and this permission notice shall be
included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/

/* String functions that mostly require knowledge of Unicode features
   of the codepoints in them. */

#include <assert.h>
#include <stdlib.h>
#include <string.h>
#include <unicode/ucnv.h>
#include <unicode/ucol.h>
#include <unicode/unorm2.h>
#include <unicode/uregex.h>
#include <unicode/uscript.h>
#include <unicode/uspoof.h>
#include <unicode/ustring.h>
#include <unicode/utf16.h>
#include <unicode/utf8.h>
#include <unicode/uversion.h>

#include <sqlite3ext.h>
SQLITE_EXTENSION_INIT1

typedef void (*version_func)(UVersionInfo);

static void icuVersionFunc(sqlite3_context *p, int nArg __attribute__((unused)),
                           sqlite3_value **apArg __attribute__((unused))) {
  UVersionInfo vers;
  char versstr[U_MAX_VERSION_STRING_LENGTH];
  version_func vf = (version_func *)sqlite3_user_data(p);

  vf(vers);

  u_versionToString(vers, versstr);
  sqlite3_result_text(p, versstr, -1, SQLITE_TRANSIENT);
}

static uint32_t parse_re_options(const unsigned char *zOptions, int *group) {
  uint32_t options = 0;
  if (zOptions) {
    while (*zOptions) {
      unsigned char c = *zOptions++;
      switch (c) {
      case 'i':
        options |= UREGEX_CASE_INSENSITIVE;
        break;
      case 'c':
        options &= ~UREGEX_CASE_INSENSITIVE;
        break;
      case 'm':
        options |= UREGEX_MULTILINE;
        break;
      case 'n':
        options |= UREGEX_DOTALL;
        break;
      case 'u':
        options |= UREGEX_UNIX_LINES;
        break;
      case 'w':
        options |= UREGEX_UWORD;
        break;
      case 'x':
        options |= UREGEX_COMMENTS;
        break;
      case 'l':
        options |= UREGEX_LITERAL;
        break;
      case '0':
      case '1':
      case '2':
      case '3':
      case '4':
      case '5':
      case '6':
      case '7':
      case '8':
      case '9':
        if (group) {
          *group = c - '0';
        }
        break;
      default:
        /* Ignore unknown flags */
        (void)0;
      }
    }
  }
  return options;
}

/* Taken straight from icu.c with very minor changes:
 *
 * regexp() takes a 3rd argument with options.
 * icu_load_collation() creates both UTF-8 and UTF-16 versions.
 * TODO: Use ICU utf-8 functions and macros?
 */

/*
** This function is called when an ICU function called from within
** the implementation of an SQL scalar function returns an error.
**
** The scalar function context passed as the first argument is
** loaded with an error message based on the following two args.
*/
void
icuFunctionError(sqlite3_context *pCtx, /* SQLite scalar function context */
                 const char *zName,     /* Name of ICU function that failed */
                 UErrorCode e /* Error code returned by ICU function */
) {
  char zBuf[128];
  sqlite3_snprintf(128, zBuf, "ICU error: %s(): %s", zName, u_errorName(e));
  zBuf[127] = '\0';
  sqlite3_result_error(pCtx, zBuf, -1);
}

/*
** Maximum length (in bytes) of the pattern in a LIKE or GLOB
** operator.
*/
#ifndef SQLITE_MAX_LIKE_PATTERN_LENGTH
#define SQLITE_MAX_LIKE_PATTERN_LENGTH 50000
#endif

/*
** Version of sqlite3_free() that is always a function, never a macro.
*/
static void xFree(void *p) { sqlite3_free(p); }

/*
** This lookup table is used to help decode the first byte of
** a multi-byte UTF8 character. It is copied here from SQLite source
** code file utf8.c.
*/
static const unsigned char icuUtf8Trans1[] = {
    0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08, 0x09, 0x0a,
    0x0b, 0x0c, 0x0d, 0x0e, 0x0f, 0x10, 0x11, 0x12, 0x13, 0x14, 0x15,
    0x16, 0x17, 0x18, 0x19, 0x1a, 0x1b, 0x1c, 0x1d, 0x1e, 0x1f, 0x00,
    0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08, 0x09, 0x0a, 0x0b,
    0x0c, 0x0d, 0x0e, 0x0f, 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06,
    0x07, 0x00, 0x01, 0x02, 0x03, 0x00, 0x01, 0x00, 0x00,
};

#define SQLITE_ICU_READ_UTF8(zIn, c)                                           \
  c = *(zIn++);                                                                \
  if (c >= 0xc0) {                                                             \
    c = icuUtf8Trans1[c - 0xc0];                                               \
    while ((*zIn & 0xc0) == 0x80) {                                            \
      c = (c << 6) + (0x3f & *(zIn++));                                        \
    }                                                                          \
  }

#define SQLITE_ICU_SKIP_UTF8(zIn)                                              \
  assert(*zIn);                                                                \
  if (*(zIn++) >= 0xc0) {                                                      \
    while ((*zIn & 0xc0) == 0x80) {                                            \
      zIn++;                                                                   \
    }                                                                          \
  }

/*
** Compare two UTF-8 strings for equality where the first string is
** a "LIKE" expression. Return true (1) if they are the same and
** false (0) if they are different.
*/
static int
icuLikeCompare(const uint8_t *zPattern, /* LIKE pattern */
               const uint8_t *zString, /* The UTF-8 string to compare against */
               const UChar32 uEsc      /* The escape character */
) {
  static const uint32_t MATCH_ONE = (uint32_t)'_';
  static const uint32_t MATCH_ALL = (uint32_t)'%';

  int prevEscape = 0; /* True if the previous character was uEsc */

  while (1) {

    /* Read (and consume) the next character from the input pattern. */
    uint32_t uPattern;
    SQLITE_ICU_READ_UTF8(zPattern, uPattern);
    if (uPattern == 0)
      break;

    /* There are now 4 possibilities:
    **
    **     1. uPattern is an unescaped match-all character "%",
    **     2. uPattern is an unescaped match-one character "_",
    **     3. uPattern is an unescaped escape character, or
    **     4. uPattern is to be handled as an ordinary character
    */
    if (!prevEscape && uPattern == MATCH_ALL) {
      /* Case 1. */
      uint8_t c;

      /* Skip any MATCH_ALL or MATCH_ONE characters that follow a
      ** MATCH_ALL. For each MATCH_ONE, skip one character in the
      ** test string.
      */
      while ((c = *zPattern) == MATCH_ALL || c == MATCH_ONE) {
        if (c == MATCH_ONE) {
          if (*zString == 0)
            return 0;
          SQLITE_ICU_SKIP_UTF8(zString);
        }
        zPattern++;
      }

      if (*zPattern == 0)
        return 1;

      while (*zString) {
        if (icuLikeCompare(zPattern, zString, uEsc)) {
          return 1;
        }
        SQLITE_ICU_SKIP_UTF8(zString);
      }
      return 0;

    } else if (!prevEscape && uPattern == MATCH_ONE) {
      /* Case 2. */
      if (*zString == 0)
        return 0;
      SQLITE_ICU_SKIP_UTF8(zString);

    } else if (!prevEscape && uPattern == (uint32_t)uEsc) {
      /* Case 3. */
      prevEscape = 1;

    } else {
      /* Case 4. */
      uint32_t uString;
      SQLITE_ICU_READ_UTF8(zString, uString);
      uString = (uint32_t)u_foldCase((UChar32)uString, U_FOLD_CASE_DEFAULT);
      uPattern = (uint32_t)u_foldCase((UChar32)uPattern, U_FOLD_CASE_DEFAULT);
      if (uString != uPattern) {
        return 0;
      }
      prevEscape = 0;
    }
  }

  return *zString == 0;
}

/*
** Implementation of the like() SQL function.  This function implements
** the build-in LIKE operator.  The first argument to the function is the
** pattern and the second argument is the string.  So, the SQL statements:
**
**       A LIKE B
**
** is implemented as like(B, A). If there is an escape character E,
**
**       A LIKE B ESCAPE E
**
** is mapped to like(B, A, E).
*/
static void icuLikeFunc(sqlite3_context *context, int argc,
                        sqlite3_value **argv) {
  const unsigned char *zA = sqlite3_value_text(argv[0]);
  const unsigned char *zB = sqlite3_value_text(argv[1]);
  UChar32 uEsc = 0;

  /* Limit the length of the LIKE or GLOB pattern to avoid problems
  ** of deep recursion and N*N behavior in patternCompare().
  */
  if (sqlite3_value_bytes(argv[0]) > SQLITE_MAX_LIKE_PATTERN_LENGTH) {
    sqlite3_result_error(context, "LIKE or GLOB pattern too complex", -1);
    return;
  }

  if (argc == 3) {
    /* The escape character string must consist of a single UTF-8 character.
    ** Otherwise, return an error.
    */
    int nE = sqlite3_value_bytes(argv[2]);
    const unsigned char *zE = sqlite3_value_text(argv[2]);
    int i = 0;
    if (zE == 0)
      return;
    U8_NEXT(zE, i, nE, uEsc);
    if (i != nE) {
      sqlite3_result_error(context,
                           "ESCAPE expression must be a single character", -1);
      return;
    }
  }

  if (zA && zB) {
    sqlite3_result_int(context, icuLikeCompare(zA, zB, uEsc));
  }
}

/*
** Function to delete compiled regexp objects. Registered as
** a destructor function with sqlite3_set_auxdata().
*/
static void icuRegexpDelete(void *p) {
  URegularExpression *pExpr = (URegularExpression *)p;
  uregex_close(pExpr);
}

/*
** Implementation of SQLite REGEXP operator. This scalar function takes
** two arguments. The first is a regular expression pattern to compile
** the second is a string to match against that pattern. If either
** argument is an SQL NULL, then NULL Is returned. Otherwise, the result
** is 1 if the string matches the pattern, or 0 otherwise.
**
** SQLite maps the regexp() function to the regexp() operator such
** that the following two are equivalent:
**
**     zString REGEXP zPattern
**     regexp(zPattern, zString)
**
** Uses the following ICU regexp APIs:
**
**     uregex_open()
**     uregex_matches()
**     uregex_close()
*/
static void icuRegexpFunc(sqlite3_context *p, int nArg, sqlite3_value **apArg) {
  UErrorCode status = U_ZERO_ERROR;
  URegularExpression *pExpr;
  UBool res;
  const UChar *zString = sqlite3_value_text16(apArg[1]);
  uint32_t options = 0;

  (void)nArg; /* Unused parameter */

  /* If the left hand side of the regexp operator is NULL,
  ** then the result is also NULL.
  */
  if (!zString) {
    return;
  }

  if (nArg == 3) {
    options = parse_re_options(sqlite3_value_text(apArg[2]), NULL);
  }

  pExpr = sqlite3_get_auxdata(p, 0);
  if (!pExpr) {
    const UChar *zPattern = sqlite3_value_text16(apArg[0]);
    if (!zPattern) {
      return;
    }
    pExpr = uregex_open(zPattern, -1, options, 0, &status);

    if (U_SUCCESS(status)) {
      sqlite3_set_auxdata(p, 0, pExpr, icuRegexpDelete);
    } else {
      assert(!pExpr);
      icuFunctionError(p, "uregex_open", status);
      return;
    }
  }

  /* Configure the text that the regular expression operates on. */
  uregex_setText(pExpr, zString, -1, &status);
  if (!U_SUCCESS(status)) {
    icuFunctionError(p, "uregex_setText", status);
    return;
  }

  /* Attempt the match */
  res = uregex_matches(pExpr, 0, &status);
  if (!U_SUCCESS(status)) {
    icuFunctionError(p, "uregex_matches", status);
    return;
  }

  /* Set the text that the regular expression operates on to a NULL
  ** pointer. This is not really necessary, but it is tidier than
  ** leaving the regular expression object configured with an invalid
  ** pointer after this function returns.
  */
  uregex_setText(pExpr, 0, 0, &status);

  /* Return 1 or 0. */
  sqlite3_result_int(p, res ? 1 : 0);
}

/*
** Implementations of scalar functions for case mapping - upper() and
** lower(). Function upper() converts its input to upper-case (ABC).
** Function lower() converts to lower-case (abc).
**
** ICU provides two types of case mapping, "general" case mapping and
** "language specific". Refer to ICU documentation for the differences
** between the two.
**
** To utilise "general" case mapping, the upper() or lower() scalar
** functions are invoked with one argument:
**
**     upper('ABC') -> 'abc'
**     lower('abc') -> 'ABC'
**
** To access ICU "language specific" case mapping, upper() or lower()
** should be invoked with two arguments. The second argument is the name
** of the locale to use. Passing an empty string ("") or SQL NULL value
** as the second argument is the same as invoking the 1 argument version
** of upper() or lower().
**
**     lower('I', 'en_us') -> 'i'
**     lower('I', 'tr_tr') -> '\u131' (small dotless i)
**
** http://www.icu-project.org/userguide/posix.html#case_mappings
*/
static void icuCaseFunc16(sqlite3_context *p, int nArg, sqlite3_value **apArg) {
  const UChar *zInput; /* Pointer to input string */
  UChar *zOutput = 0;  /* Pointer to output buffer */
  int nInput;          /* Size of utf-16 input string in bytes */
  int nOut;            /* Size of output buffer in bytes */
  int cnt;
  int bToUpper; /* True for toupper(), false for tolower() */
  UErrorCode status;
  const char *zLocale = 0;

  assert(nArg == 1 || nArg == 2);
  bToUpper = (sqlite3_user_data(p) != 0);
  if (nArg == 2) {
    zLocale = (const char *)sqlite3_value_text(apArg[1]);
  }

  zInput = sqlite3_value_text16(apArg[0]);
  if (!zInput) {
    return;
  }
  nOut = nInput = sqlite3_value_bytes16(apArg[0]);
  if (nOut == 0) {
    sqlite3_result_text16(p, "", 0, SQLITE_STATIC);
    return;
  }

  for (cnt = 0; cnt < 2; cnt++) {
    UChar *zNew = sqlite3_realloc(zOutput, nOut);
    if (zNew == 0) {
      sqlite3_free(zOutput);
      sqlite3_result_error_nomem(p);
      return;
    }
    zOutput = zNew;
    status = U_ZERO_ERROR;
    if (bToUpper) {
      nOut = 2 * u_strToUpper(zOutput, nOut / 2, zInput, nInput / 2, zLocale,
                              &status);
    } else {
      nOut = 2 * u_strToLower(zOutput, nOut / 2, zInput, nInput / 2, zLocale,
                              &status);
    }

    if (U_SUCCESS(status)) {
      sqlite3_result_text16(p, zOutput, nOut, xFree);
    } else if (status == U_BUFFER_OVERFLOW_ERROR) {
      assert(cnt == 0);
      continue;
    } else {
      icuFunctionError(p, bToUpper ? "u_strToUpper" : "u_strToLower", status);
    }
    return;
  }
  assert(0); /* Unreachable */
}

/*
** Collation sequence destructor function. The pCtx argument points to
** a UCollator structure previously allocated using ucol_open().
*/
static void icuCollationDel(void *pCtx) {
  UCollator *p = (UCollator *)pCtx;
  ucol_close(p);
}

/*
** Collation sequence comparison function using UTF-16. The pCtx
* argument points to a UCollator structure previously allocated
* using ucol_open().
*/
static int icuCollationColl16(void *pCtx, int nLeft, const void *zLeft,
                              int nRight, const void *zRight) {
  UCollationResult res;
  UCollator *p = (UCollator *)pCtx;
  res = ucol_strcoll(p, (const UChar *)zLeft, nLeft / 2, (const UChar *)zRight,
                     nRight / 2);
  switch (res) {
  case UCOL_LESS:
    return -1;
  case UCOL_GREATER:
    return +1;
  case UCOL_EQUAL:
    return 0;
  }
  assert(!"Unexpected return value from ucol_strcoll()");
  return 0;
}

/*
** Collation sequence comparison function using UTF-8. The pCtx
* argument points to a UCollator structure previously allocated using
* ucol_open().
*/
static int icuCollationColl8(void *pCtx, int nLeft, const void *zLeft,
                             int nRight, const void *zRight) {
  UErrorCode status = U_ZERO_ERROR;
  UCollationResult res;
  UCollator *p = (UCollator *)pCtx;
  res = ucol_strcollUTF8(p, (const char *)zLeft, nLeft, (const char *)zRight,
                         nRight, &status);
  assert(U_SUCCESS(status));
  switch (res) {
  case UCOL_LESS:
    return -1;
  case UCOL_GREATER:
    return +1;
  case UCOL_EQUAL:
    return 0;
  }
  assert(!"Unexpected return value from ucol_strcollUTF8()");
  return 0;
}

/*
** Implementation of the scalar function icu_load_collation().
**
** This scalar function is used to add ICU collation based collation
** types to an SQLite database connection. It is intended to be called
** as follows:
**
**     SELECT icu_load_collation(<locale>, <collation-name>);
**
** Where <locale> is a string containing an ICU locale identifier (i.e.
** "en_AU", "tr_TR" etc.) and <collation-name> is the name of the
** collation sequence to create.
*/
static void icuLoadCollation(sqlite3_context *p,
                             int nArg __attribute__((unused)),
                             sqlite3_value **apArg) {
  sqlite3 *db = (sqlite3 *)sqlite3_user_data(p);
  UErrorCode status = U_ZERO_ERROR;
  const char *zLocale;   /* Locale identifier - (eg. "jp_JP") */
  const char *zName;     /* SQL Collation sequence name (eg. "japanese") */
  UCollator *pUCollator; /* ICU library collation object */
  int rc;                /* Return code from sqlite3_create_collation_x() */

  assert(nArg == 2);
  zLocale = (const char *)sqlite3_value_text(apArg[0]);
  zName = (const char *)sqlite3_value_text(apArg[1]);

  if (!zLocale || !zName) {
    return;
  }

  pUCollator = ucol_open(zLocale, &status);
  if (U_FAILURE(status)) {
    icuFunctionError(p, "ucol_open", status);
    return;
  }
  assert(pUCollator);

  UCollator *pUCollator2 = ucol_safeClone(pUCollator, NULL, NULL, &status);
  if (U_FAILURE(status)) {
    icuFunctionError(p, "ucol_safeClone", status);
    ucol_close(pUCollator);
    return;
  }
  assert(pUCollator2);

  rc = sqlite3_create_collation_v2(db, zName, SQLITE_UTF16, (void *)pUCollator,
                                   icuCollationColl16, icuCollationDel);
  if (rc != SQLITE_OK) {
    sqlite3_result_error(p, "Error registering collation function", -1);
    ucol_close(pUCollator2);
    return;
  }

  rc = sqlite3_create_collation_v2(db, zName, SQLITE_UTF8, (void *)pUCollator2,
                                   icuCollationColl8, icuCollationDel);
  if (rc != SQLITE_OK) {
    ucol_close(pUCollator2);
    sqlite3_result_error(p, "Error registering collation function", -1);
  }
}

/* Back to original stuff */

/* Based on icuCaseFunc16 */
static void icuTitleFunc16(sqlite3_context *p, int nArg,
                           sqlite3_value **apArg) {
  const UChar *zInput; /* Pointer to input string */
  UChar *zOutput = 0;  /* Pointer to output buffer */
  int nInput;          /* Size of utf-16 input string in bytes */
  sqlite3_uint64 nOut; /* Size of output buffer in bytes */
  UErrorCode status;
  const char *zLocale = 0;

  assert(nArg == 1 || nArg == 2);
  if (nArg == 2) {
    zLocale = (const char *)sqlite3_value_text(apArg[1]);
  }

  zInput = sqlite3_value_text16(apArg[0]);
  if (!zInput) {
    return;
  }
  nOut = nInput = sqlite3_value_bytes16(apArg[0]);
  if (nInput == 0) {
    sqlite3_result_text16(p, "", 0, SQLITE_STATIC);
    return;
  }

  for (int cnt = 0; cnt < 2; cnt++) {
    UChar *zNew = sqlite3_realloc64(zOutput, nOut);
    if (zNew == 0) {
      sqlite3_free(zOutput);
      sqlite3_result_error_nomem(p);
      return;
    }
    zOutput = zNew;
    status = U_ZERO_ERROR;
    nOut = (sqlite3_uint64)2 * u_strToTitle(zOutput, nOut / 2, zInput,
                                            nInput / 2, NULL, zLocale, &status);

    if (U_SUCCESS(status)) {
      sqlite3_result_text64(p, (char *)zOutput, nOut, sqlite3_free,
                            SQLITE_UTF16);
    } else if (status == U_BUFFER_OVERFLOW_ERROR) {
      assert(cnt == 0);
      continue;
    } else {
      icuFunctionError(p, "u_strToTitle", status);
      sqlite3_free(zOutput);
    }
    return;
  }
  assert(0); /* Unreachable */
}

/* Basec on icuCaseFunc16 */
static void icuCaseFoldFunc16(sqlite3_context *p,
                              int nArg __attribute__((unused)),
                              sqlite3_value **apArg) {
  const UChar *zInput; /* Pointer to input string */
  UChar *zOutput = 0;  /* Pointer to output buffer */
  int nInput;          /* Size of utf-16 input string in bytes */
  sqlite3_uint64 nOut; /* Size of output buffer in bytes */
  UErrorCode status;

  assert(nArg == 1);

  zInput = sqlite3_value_text16(apArg[0]);
  if (!zInput) {
    return;
  }
  nOut = nInput = sqlite3_value_bytes16(apArg[0]);
  if (nInput == 0) {
    sqlite3_result_text16(p, "", 0, SQLITE_STATIC);
    return;
  }

  for (int cnt = 0; cnt < 2; cnt++) {
    UChar *zNew = sqlite3_realloc64(zOutput, nOut);
    if (zNew == 0) {
      sqlite3_free(zOutput);
      sqlite3_result_error_nomem(p);
      return;
    }
    zOutput = zNew;
    status = U_ZERO_ERROR;
    nOut =
        (sqlite3_uint64)2 * u_strFoldCase(zOutput, nOut / 2, zInput, nInput / 2,
                                          U_FOLD_CASE_DEFAULT, &status);

    if (U_SUCCESS(status)) {
      sqlite3_result_text64(p, (char *)zOutput, nOut, sqlite3_free,
                            SQLITE_UTF16);
    } else if (status == U_BUFFER_OVERFLOW_ERROR) {
      assert(cnt == 0);
      continue;
    } else {
      icuFunctionError(p, "u_strFoldCase", status);
      sqlite3_free(zOutput);
    }
    return;
  }
  assert(0); /* Unreachable */
}

typedef const UNormalizer2 *(*norm_func)(UErrorCode *);
static struct normalization_forms {
  const char *name;
  norm_func f;
  const char *fname;
} normalizers[] = {{"NFC", unorm2_getNFCInstance, "unorm2_getNFCInstance"},
                   {"NFD", unorm2_getNFDInstance, "unorm2_getNFDInstance"},
                   {"NFKC", unorm2_getNFKCInstance, "unorm2_getNFKCInstance"},
                   {"NFKD", unorm2_getNFKDInstance, "unorm2_getNFKDInstance"},
                   {"NFKCCaseFold", unorm2_getNFKCCasefoldInstance,
                    "unorm2_getNFKCCasefoldInstance"},
                   {NULL, NULL, NULL}};

static const UNormalizer2 *icu_getNormForm(sqlite3_context *ctx,
                                           const char *form) {
  UErrorCode status = U_ZERO_ERROR;
  if (form) {
    for (int n = 0; normalizers[n].name; n += 1) {
      if (sqlite3_stricmp(normalizers[n].name, form) == 0) {
        const UNormalizer2 *norm = normalizers[n].f(&status);
        if (U_FAILURE(status)) {
          icuFunctionError(ctx, normalizers[n].fname, status);
          return NULL;
        }
        return norm;
      }
    }
  }
  sqlite3_result_error(ctx, "Invalid normalization form.", -1);
  return NULL;
}

static UChar *icuNormalizeUChar(sqlite3_context *p, const UNormalizer2 *norm,
                                const UChar *zInput, int nInput,
                                sqlite3_uint64 *nOutput) {
  UErrorCode status = U_ZERO_ERROR;
  UChar *zOut = NULL;
  sqlite3_uint64 nOut = nInput;

  if (nInput == 0) {
    zOut = sqlite3_malloc(2);
    if (!zOut) {
      sqlite3_result_error_nomem(p);
      return NULL;
    }
    *zOut = 0;
    if (nOutput) {
      *nOutput = 0;
    }
    return zOut;
  }

  if (unorm2_quickCheck(norm, zInput, nInput / 2, &status) == UNORM_YES &&
      U_SUCCESS(status)) {
    // fast path for an already appropriately normalized string
    zOut = sqlite3_malloc(nInput);
    if (!zOut) {
      sqlite3_result_error_nomem(p);
      return NULL;
    }
    memcpy(zOut, zInput, nInput);
    if (nOutput) {
      *nOutput = nInput;
    }
    return zOut;
  } else {
    for (int cnt = 0; cnt < 2; cnt++) {
      UChar *zNew = sqlite3_realloc64(zOut, nOut);
      if (zNew == 0) {
        sqlite3_free(zOut);
        sqlite3_result_error_nomem(p);
        return NULL;
      }
      zOut = zNew;
      status = U_ZERO_ERROR;
      nOut = (sqlite3_uint64)2 * unorm2_normalize(norm, zInput, nInput / 2,
                                                  zOut, nOut / 2, &status);
      if (U_SUCCESS(status)) {
        if (nOutput) {
          *nOutput = nOut;
        }
        return zOut;
      } else if (status == U_BUFFER_OVERFLOW_ERROR) {
        assert(cnt == 0);
        continue;
      } else {
        icuFunctionError(p, "unorm2_normalize", status);
        sqlite3_free(zOut);
        return NULL;
      }
    }
  }
  assert(0);
  return NULL;
}

// Normalize a string
static void icuNormFunc16(sqlite3_context *p, int nArg __attribute__((unused)),
                          sqlite3_value **apArg) {
  const UChar *zInput; /* Pointer to input string */
  UChar *zOutput = 0;  /* Pointer to output buffer */
  int nInput;          /* Size of utf-16 input string in bytes */
  sqlite3_uint64 nOut; /* Size of output buffer in bytes */

  assert(nArg == 2);

  const char *form = (const char *)sqlite3_value_text(apArg[1]);

  const UNormalizer2 *norm = icu_getNormForm(p, form);
  if (!norm) {
    return;
  }

  zInput = sqlite3_value_text16(apArg[0]);
  if (!zInput) {
    return;
  }
  nInput = sqlite3_value_bytes16(apArg[0]);
  if (nInput == 0) {
    sqlite3_result_text(p, "", 0, SQLITE_STATIC);
    return;
  }

  zOutput = icuNormalizeUChar(p, norm, zInput, nInput, &nOut);
  if (zOutput) {
    sqlite3_result_text64(p, (char *)zOutput, nOut, sqlite3_free, SQLITE_UTF16);
  }
}

// Append two normalized strings
static UChar *icuAppendNormUChars(sqlite3_context *p, const UNormalizer2 *norm,
                                  UChar *zOutput, sqlite3_uint64 *nOutput,
                                  const UChar *zApp, int nApp) {
  UErrorCode status = U_ZERO_ERROR;
  sqlite3_uint64 nOut = *nOutput;
  sqlite3_uint64 nNewLen = nOut + nApp;

  if (nApp == 0) {
    return zOutput;
  }

  for (int cnt = 0; cnt < 2; cnt++) {
    UChar *zNew = sqlite3_realloc64(zOutput, nNewLen);
    if (zNew == 0) {
      sqlite3_free(zOutput);
      sqlite3_result_error_nomem(p);
      return NULL;
    }
    zOutput = zNew;
    status = U_ZERO_ERROR;
    nNewLen =
        (sqlite3_uint64)2 * unorm2_append(norm, zOutput, nOut / 2, nNewLen / 2,
                                          zApp, nApp / 2, &status);
    if (U_SUCCESS(status)) {
      *nOutput = nNewLen;
      return zOutput;
    } else if (status == U_BUFFER_OVERFLOW_ERROR) {
      assert(cnt == 0);
      continue;
    } else {
      icuFunctionError(p, "unorm2_append", status);
      return NULL;
    }
  }
  assert(0);
  return NULL;
}

// Append an unnormal string to a normalized one
static UChar *icuAppendUnNormUChars(sqlite3_context *p,
                                    const UNormalizer2 *norm, UChar *zOutput,
                                    sqlite3_uint64 *nOutput, const UChar *zApp,
                                    int nApp) {
  UErrorCode status = U_ZERO_ERROR;
  sqlite3_uint64 nOut = *nOutput;
  sqlite3_uint64 nNewLen = nOut + nApp;

  for (int cnt = 0; cnt < 2; cnt++) {
    UChar *zNew = sqlite3_realloc64(zOutput, nNewLen);
    if (zNew == 0) {
      sqlite3_free(zOutput);
      sqlite3_result_error_nomem(p);
      return NULL;
    }
    zOutput = zNew;
    status = U_ZERO_ERROR;
    nNewLen = (sqlite3_uint64)2 * unorm2_normalizeSecondAndAppend(
                                      norm, zOutput, nOut / 2, nNewLen / 2,
                                      zApp, nApp / 2, &status);
    if (U_SUCCESS(status)) {
      *nOutput = nNewLen;
      return zOutput;
    } else if (status == U_BUFFER_OVERFLOW_ERROR) {
      assert(cnt == 0);
      continue;
    } else {
      icuFunctionError(p, "unorm2_normalizeSecondAndAppend", status);
      return NULL;
    }
  }
  assert(0);
  return NULL;
}

static void icuNormConcatFunc16(sqlite3_context *p, int nArg,
                                sqlite3_value **apArg) {
  const UNormalizer2 *norm = sqlite3_user_data(p);
  UChar *zOutput = NULL;
  sqlite3_uint64 nOut = 0;
  _Bool empty = 0;
  UErrorCode status;

  for (int n = 0; n < nArg; n += 1) {
    if (sqlite3_value_type(apArg[n]) == SQLITE_NULL) {
      continue;
    }
    int nInput;
    const UChar *zInput = sqlite3_value_text16(apArg[n]);
    if (!zInput) {
      continue;
    }
    nInput = sqlite3_value_bytes16(apArg[n]);
    if (nInput == 0) {
      empty = 1;
      continue;
    }
    if (zOutput) {
      if (unorm2_quickCheck(norm, zInput, nInput / 2, &status) == UNORM_YES &&
          U_SUCCESS(status)) {
        zOutput = icuAppendNormUChars(p, norm, zOutput, &nOut, zInput, nInput);
        if (!zOutput) {
          return;
        }
      } else {
        // String to append isn't normalized
        zOutput =
            icuAppendUnNormUChars(p, norm, zOutput, &nOut, zInput, nInput);
        if (!zOutput) {
          return;
        }
      }
    } else {
      // First non-null argument
      zOutput = icuNormalizeUChar(p, norm, zInput, nInput, &nOut);
      if (!zOutput) {
        return;
      }
    }
  }

  if (zOutput) {
    sqlite3_result_text64(p, (char *)zOutput, nOut, sqlite3_free, SQLITE_UTF16);
  } else if (empty) {
    sqlite3_result_text(p, "", 0, SQLITE_STATIC);
  }
}

static void icuRepeatFunc16(sqlite3_context *p,
                            int nArg __attribute__((unused)),
                            sqlite3_value **apArg) {
  assert(nArg == 3);

  if (sqlite3_value_type(apArg[0]) == SQLITE_NULL ||
      sqlite3_value_type(apArg[1]) == SQLITE_NULL) {
    return;
  }

  const char *form = (const char *)sqlite3_value_text(apArg[2]);
  const UNormalizer2 *norm = icu_getNormForm(p, form);
  if (!norm) {
    return;
  }

  int reps = sqlite3_value_int(apArg[1]);
  if (reps <= 0) {
    return;
  }

  const UChar *utf16 = sqlite3_value_text16(apArg[0]);
  if (!utf16) {
    return;
  }

  int len = sqlite3_value_bytes16(apArg[0]);

  sqlite3_uint64 nlen;
  UChar *normed = icuNormalizeUChar(p, norm, utf16, len, &nlen);
  if (!normed) {
    return;
  }

  UChar *zOut = sqlite3_malloc64(nlen);
  sqlite3_uint64 nOut = nlen;
  if (!zOut) {
    sqlite3_free(normed);
    sqlite3_result_error_nomem(p);
  }
  memcpy(zOut, normed, nlen);
  reps -= 1;

  while (reps--) {
    UChar *zNew = icuAppendNormUChars(p, norm, zOut, &nOut, normed, nlen);
    if (!zNew) {
      sqlite3_free(normed);
      return;
    }
    zOut = zNew;
  }
  sqlite3_free(normed);
  sqlite3_result_text64(p, (char *)zOut, nOut, sqlite3_free, SQLITE_UTF16);
}

static void icuNormConcatWSFunc16(sqlite3_context *p, int nArg,
                                  sqlite3_value **apArg) {
  const UNormalizer2 *norm = sqlite3_user_data(p);
  UChar *zOutput = NULL;
  sqlite3_uint64 nOut = 0;
  UErrorCode status;
  UChar *zSep = NULL;
  sqlite3_uint64 nSep;

  if (nArg <= 1) {
    return;
  }

  if (sqlite3_value_type(apArg[0]) == SQLITE_NULL) {
    return;
  }

  {
    const UChar *sep = sqlite3