API for label block data type (github.com/janelia-flyem/dvid/datatype/labelarray) =============================================================================== Note: UUIDs referenced below are strings that may either be a unique prefix of a hexadecimal UUID string (e.g., 3FA22) or a branch leaf specification that adds a colon (":") followed by the case-dependent branch name. In the case of a branch leaf specification, the unique UUID prefix just identifies the repo of the branch, and the UUID referenced is really the leaf of the branch name. For example, if we have a DAG with root A -> B -> C where C is the current HEAD or leaf of the "master" (default) branch, then asking for "B:master" is the same as asking for "C". If we add another version so A -> B -> C -> D, then references to "B:master" now return the data from "D". --- Denormalizations like sparse volumes are *not* performed for the "0" label, which is considered a special label useful for designating background. This allows users to define sparse labeled structures in a large volume without requiring processing of entire volume. Command-line: $ dvid repo new labelarray Adds newly named data of the 'type name' to repo with specified UUID. Example (note anisotropic resolution specified instead of default 8 nm isotropic): $ dvid repo 3f8c new labelarray superpixels VoxelSize=3.2,3.2,40.0 Arguments: UUID Hexadecimal string with enough characters to uniquely identify a version node. data name Name of data to create, e.g., "superpixels" settings Configuration settings in "key=value" format separated by spaces. Configuration Settings (case-insensitive keys) BlockSize Size in pixels (default: %!s(MISSING)) VoxelSize Resolution of voxels (default: 8.0, 8.0, 8.0) VoxelUnits Resolution units (default: "nanometers") IndexedLabels "false" if no sparse volume support is required (default "true") CountLabels "false" if no voxel counts per label is required (default "true") MaxDownresLevel The maximum down-res level supported. Each down-res is factor of 2. $ dvid node load Initializes version node to a set of XY label images described by glob of filenames. The DVID server must have access to the named files. Currently, XY images are required. Example: $ dvid node 3f8c superpixels load 0,0,100 "data/*.png" proc=noindex Arguments: UUID Hexadecimal string with enough characters to uniquely identify a version node. data name Name of data to add. offset 3d coordinate in the format "x,y,z". Gives coordinate of top upper left voxel. image glob Filenames of label images, preferably in quotes, e.g., "foo-xy-*.png" $ dvid node composite Creates a RGBA8 image where the RGB is a hash of the labels and the A is the grayscale intensity. Example: $ dvid node 3f8c bodies composite grayscale bodyview Arguments: UUID Hexadecimal string with enough characters to uniquely identify a version node. data name Name of data to add. ------------------ HTTP API (Level 2 REST): POST /api/repo/{uuid}/instance Creates a new instance of the labelarray data type. Expects configuration data in JSON as the body of the POST. Configuration data is a JSON object with each property corresponding to a configuration keyword for the particular data type. JSON name/value pairs: REQUIRED "typename" Must be "labelarray" REQUIRED "dataname" Name of the new instance OPTIONAL "versioned" If "false" or "0", the data is unversioned and acts as if all UUIDs within a repo become the root repo UUID. (True by default.) OPTIONAL "BlockSize" Size in pixels (default: 64,64,64) OPTIONAL "VoxelSize" Resolution of voxels (default: 8.0,8.0,8.0) OPTIONAL "VoxelUnits" Resolution units (default: "nanometers") OPTIONAL "IndexedLabels" "false" if no sparse volume support is required (default "true") OPTIONAL "CountLabels" "false" if no voxel counts per label is required (default "true") OPTIONAL "MaxDownresLevel" The maximum down-res level supported. Each down-res is factor of 2. GET /node///help Returns data-specific help message. GET /node///info POST /node///info Retrieves or puts DVID-specific data properties for these voxels. Example: GET /node/3f8c/segmentation/info Returns JSON with configuration settings that include location in DVID space and min/max block indices. Arguments: UUID Hexadecimal string with enough characters to uniquely identify a version node. data name Name of voxels data. POST /node///resolution Sets the resolution for the image volume. Extents should be in JSON in the following format: [8,8,8] POST /node///sync? Establishes labelvol data instances with which the annotations are synced. Expects JSON to be POSTed with the following format: { "sync": "bodies" } To delete syncs, pass an empty string of names with query string "replace=true": { "sync": "" } The "sync" property should be followed by a comma-delimited list of data instances that MUST already exist. Currently, syncs should be created before any annotations are pushed to the server. If annotations already exist, these are currently not synced. The labelarray data type accepts syncs to labelvol data instances. It also accepts syncs to labelarray instances for multiscale. GET Query-string Options: replace Set to "true" if you want passed syncs to replace and not be appended to current syncs. Default operation is false. GET /node///metadata Retrieves a JSON schema (application/vnd.dvid-nd-data+json) that describes the layout of bytes returned for n-d images. GET /node///specificblocks[?queryopts] Retrieves blocks corresponding to those specified in the query string. This interface is useful if the blocks retrieved are not consecutive or if the backend in non ordered. TODO: enable arbitrary compression to be specified Example: GET /node/3f8c/grayscale/specificblocks?blocks=x1,y1,z2,x2,y2,z2,x3,y3,z3 This will fetch blocks at position (x1,y1,z1), (x2,y2,z2), and (x3,y3,z3). The returned byte stream has a list of blocks with a leading block coordinate (3 x int32) plus int32 giving the # of bytes in this block, and then the bytes for the value. If blocks are unset within the span, they will not appear in the stream, so the returned data will be equal to or less than spanX blocks worth of data. The returned data format has the following format where int32 is in little endian and the bytes of block data have been compressed in JPEG format. int32 Block 1 coordinate X (Note that this may not be starting block coordinate if it is unset.) int32 Block 1 coordinate Y int32 Block 1 coordinate Z int32 # bytes for first block (N1) byte0 Bytes of block data in jpeg-compressed format. byte1 ... byteN1 int32 Block 2 coordinate X int32 Block 2 coordinate Y int32 Block 2 coordinate Z int32 # bytes for second block (N2) byte0 Bytes of block data in jpeg-compressed format. byte1 ... byteN2 ... If no data is available for given block span, nothing is returned. Arguments: UUID Hexadecimal string with enough characters to uniquely identify a version node. data name Name of data to add. Query-string Options: blocks x,y,z... block string scale A number from 0 up to MaxDownresLevel where each level has 1/2 resolution of previous level. Level 0 (default) is the highest resolution. GET /node///isotropic///[/][?queryopts] Retrieves either 2d images (PNG by default) or 3d binary data, depending on the dims parameter. The 3d binary data response has "Content-type" set to "application/octet-stream" and is an array of voxel values in ZYX order (X iterates most rapidly). Example: GET /node/3f8c/segmentation/isotropic/0_1/512_256/0_0_100/jpg:80 Returns an isotropic XY slice (0th and 1st dimensions) with width (x) of 512 voxels and height (y) of 256 voxels with offset (0,0,100) in JPG format with quality 80. Additional processing is applied based on voxel resolutions to make sure the retrieved image has isotropic pixels. For example, if an XZ image is requested and the image volume has X resolution 3 nm and Z resolution 40 nm, the returned image's height will be magnified 40/3 relative to the raw data. The example offset assumes the "grayscale" data in version node "3f8c" is 3d. The "Content-type" of the HTTP response should agree with the requested format. For example, returned PNGs will have "Content-type" of "image/png", and returned nD data will be "application/octet-stream". Arguments: UUID Hexadecimal string with enough characters to uniquely identify a version node. data name Name of data to add. dims The axes of data extraction in form "i_j_k,..." Example: "0_2" can be XZ. Slice strings ("xy", "xz", or "yz") are also accepted. size Size in voxels along each dimension specified in . offset Gives coordinate of first voxel using dimensionality of data. format Valid formats depend on the dimensionality of the request and formats available in server implementation. 2D: "png", "jpg" (default: "png") jpg allows lossy quality setting, e.g., "jpg:80" nD: uses default "octet-stream". Query-string Options: roi Name of roi data instance used to mask the requested data. scale A number from 0 up to MaxDownresLevel where each level beyond 0 has 1/2 resolution of previous level. Level 0 is the highest resolution. compression Allows retrieval or submission of 3d data in "lz4" and "gzip" compressed format. The 2d data will ignore this and use the image-based codec. throttle Only works for 3d data requests. If "true", makes sure only N compute-intense operation (all API calls that can be throttled) are handled. If the server can't initiate the API call right away, a 503 (Service Unavailable) status code is returned. GET /node///raw///[/][?queryopts] Retrieves either 2d images (PNG by default) or 3d binary data, depending on the dims parameter. The 3d binary data response has "Content-type" set to "application/octet-stream" and is an array of voxel values in ZYX order (X iterates most rapidly). Example: GET /node/3f8c/segmentation/raw/0_1/512_256/0_0_100/jpg:80 Returns a raw XY slice (0th and 1st dimensions) with width (x) of 512 voxels and height (y) of 256 voxels with offset (0,0,100) in JPG format with quality 80. By "raw", we mean that no additional processing is applied based on voxel resolutions to make sure the retrieved image has isotropic pixels. The example offset assumes the "grayscale" data in version node "3f8c" is 3d. The "Content-type" of the HTTP response should agree with the requested format. For example, returned PNGs will have "Content-type" of "image/png", and returned nD data will be "application/octet-stream". Arguments: UUID Hexadecimal string with enough characters to uniquely identify a version node. data name Name of data to add. dims The axes of data extraction in form "i_j_k,..." Slice strings ("xy", "xz", or "yz") are also accepted. Example: "0_2" is XZ, and "0_1_2" is a 3d subvolume. size Size in voxels along each dimension specified in . offset Gives coordinate of first voxel using dimensionality of data. format Valid formats depend on the dimensionality of the request and formats available in server implementation. 2D: "png", "jpg" (default: "png") jpg allows lossy quality setting, e.g., "jpg:80" nD: uses default "octet-stream". Query-string Options: roi Name of roi data instance used to mask the requested data. scale A number from 0 up to MaxDownresLevel where each level beyond 0 has 1/2 resolution of previous level. Level 0 is the highest resolution. compression Allows retrieval or submission of 3d data in "lz4","gzip", "google" (neuroglancer compression format), "googlegzip" (google + gzip) compressed format. The 2d data will ignore this and use the image-based codec. throttle Only works for 3d data requests. If "true", makes sure only N compute-intense operation (all API calls that can be throttled) are handled. If the server can't initiate the API call right away, a 503 (Service Unavailable) status code is returned. POST /node///raw/0_1_2//[?queryopts] Puts block-aligned voxel data using the block sizes defined for this data instance. For example, if the BlockSize = 32, offset and size must be multiples of 32. Example: POST /node/3f8c/segmentation/raw/0_1_2/512_256_128/0_0_32 Arguments: UUID Hexadecimal string with enough characters to uniquely identify a version node. data name Name of data to add. size Size in voxels along each dimension specified in . offset Gives coordinate of first voxel using dimensionality of data. Query-string Options: roi Name of roi data instance used to mask the requested data. mutate Default "false" corresponds to ingestion, i.e., the first write of the given block. Use "true" to indicate the POST is a mutation of prior data, which allows any synced data instance to cleanup prior denormalizations. If "mutate=true", the POST operations will be slower due to a required GET to retrieve past data. compression Allows retrieval or submission of 3d data in "lz4" and "gzip" compressed format. throttle If "true", makes sure only N compute-intense operation (all API calls that can be throttled) are handled. If the server can't initiate the API call right away, a 503 (Service Unavailable) status code is returned. GET /node///pseudocolor///[?queryopts] Retrieves label data as pseudocolored 2D PNG color images where each label hashed to a different RGB. Example: GET /node/3f8c/segmentation/pseudocolor/0_1/512_256/0_0_100 Returns an XY slice (0th and 1st dimensions) with width (x) of 512 voxels and height (y) of 256 voxels with offset (0,0,100) in PNG format. Arguments: UUID Hexadecimal string with enough characters to uniquely identify a version node. data name Name of data to add. dims The axes of data extraction. Example: "0_2" can be XZ. Slice strings ("xy", "xz", or "yz") are also accepted. size Size in voxels along each dimension specified in . offset Gives coordinate of first voxel using dimensionality of data. Query-string Options: roi Name of roi data instance used to mask the requested data. compression Allows retrieval or submission of 3d data in "lz4" and "gzip" compressed format. throttle If "true", makes sure only N compute-intense operation (all API calls that can be throttled) are handled. If the server can't initiate the API call right away, a 503 (Service Unavailable) status code is returned. GET /node///label/[?queryopts] Returns JSON for the label at the given coordinate: { "Label": 23 } Arguments: UUID Hexadecimal string with enough characters to uniquely identify a version node. data name Name of label data. coord Coordinate of voxel with underscore as separator, e.g., 10_20_30 Query-string Options: scale A number from 0 up to MaxDownresLevel where each level beyond 0 has 1/2 resolution of previous level. Level 0 is the highest resolution. GET /node///labels[?queryopts] Returns JSON for the labels at a list of coordinates. Expects JSON in GET body: [ [x0, y0, z0], [x1, y1, z1], ...] Returns for each POSTed coordinate the corresponding label: [ 23, 911, ...] Arguments: UUID Hexadecimal string with enough characters to uniquely identify a version node. data name Name of label data. Query-string Options: scale A number from 0 up to MaxDownresLevel where each level beyond 0 has 1/2 resolution of previous level. Level 0 is the highest resolution. hash MD5 hash of request body content in hexidecimal string format. GET /node///blocks//[?queryopts] Gets blocks corresponding to the extents specified by the size and offset. The subvolume request must be block aligned. This is the most server-efficient way of retrieving the labelarray data, where data read from the underlying storage engine is written directly to the HTTP connection possibly after recompression to match the given query-string compression option. The default labelarray compression is gzip on compressed DVID label Block serialization ("blocks" option). Example: GET /node/3f8c/segmentation/blocks/64_64_64/0_0_0 If block size is 32x32x32, this call retrieves up to 8 blocks where the first potential block is at 0, 0, 0. The returned byte stream has a list of blocks with a leading block coordinate (3 x int32) plus int32 giving the # of bytes in this block, and then the bytes for the value. If blocks are unset within the span, they will not appear in the stream, so the returned data will be equal to or less than spanX blocks worth of data. The returned data format has the following format where int32 is in little endian and the bytes of block data have been compressed in the desired output format. int32 Block 1 coordinate X (Note that this may not be starting block coordinate if it is unset.) int32 Block 1 coordinate Y int32 Block 1 coordinate Z int32 # bytes for first block (N1) byte0 Block N1 serialization using chosen compression format (see "compression" option below) byte1 ... byteN1 int32 Block 2 coordinate X int32 Block 2 coordinate Y int32 Block 2 coordinate Z int32 # bytes for second block (N2) byte0 Block N2 serialization using chosen compression format (see "compression" option below) byte1 ... byteN2 ... If no data is available for given block span, nothing is returned. Arguments: UUID Hexadecimal string with enough characters to uniquely identify a version node. data name Name of data to add. size Size in voxels along each dimension specified in . offset Gives coordinate of first voxel using dimensionality of data. Query-string Options: scale A number from 0 up to MaxDownresLevel where each level beyond 0 has 1/2 resolution of previous level. Level 0 is the highest resolution. compression Allows retrieval of block data in "lz4" (default), "gzip", blocks" (native DVID label blocks) or "uncompressed" (uint64 labels). throttle If "true", makes sure only N compute-intense operation (all API calls that can be throttled) are handled. If the server can't initiate the API call right away, a 503 (Service Unavailable) status code is returned. POST /node///blocks[?queryopts] Puts properly-sized blocks for this data instance. This is the most server-efficient way of storing labelarray data, where data read from the HTTP stream is written directly to the underlying storage. The default (and currently only supported) compression is gzip on compressed DVID label Block serialization. Note that maximum label and extents are automatically handled during these calls. Example: POST /node/3f8c/segmentation/blocks The posted data format should be in the following format where int32 is in little endian and the bytes of block data have been compressed in the desired output format. int32 Block 1 coordinate X (Note that this may not be starting block coordinate if it is unset.) int32 Block 1 coordinate Y int32 Block 1 coordinate Z int32 # bytes for first block (N1) byte0 Block N1 serialization using chosen compression format (see "compression" option below) byte1 ... byteN1 int32 Block 2 coordinate X int32 Block 2 coordinate Y int32 Block 2 coordinate Z int32 # bytes for second block (N2) byte0 Block N2 serialization using chosen compression format (see "compression" option below) byte1 ... byteN2 ... The Block serialization format is as follows: 3 * uint32 values of gx, gy, and gz uint32 # of labels (N), cannot exceed uint32. N * uint64 packed labels in little-endian format. Label 0 can be used to represent deleted labels, e.g., after a merge operation to avoid changing all sub-block indices. ----- Data below is only included if N > 1, otherwise it is a solid block. Nsb = # sub-blocks = gx * gy * gz Nsb * uint16 # of labels for sub-blocks. Each uint16 Ns[i] = # labels for sub-block i. If Ns[i] == 0, the sub-block has no data (uninitialized), which is useful for constructing Blocks with sparse data. Nsb * Ns * uint32 label indices for sub-blocks where Ns = sum of Ns[i] over all sub-blocks. For each sub-block i, we have Ns[i] label indices of lBits. Nsb * values sub-block indices for each voxel. Data encompasses 512 * ceil(log2(Ns[i])) bits, padded so no two sub-blocks have indices in the same byte. At most we use 9 bits per voxel for up to the 512 labels in sub-block. A value gives the sub-block index which points to the index into the N labels. If Ns[i] <= 1, there are no values. If Ns[i] = 0, the 8x8x8 voxels are set to label 0. If Ns[i] = 1, all voxels are the given label index. Arguments: UUID Hexadecimal string with enough characters to uniquely identify a version node. data name Name of data to add. Query-string Options: scale A number from 0 up to MaxDownresLevel where each level beyond 0 has 1/2 resolution of previous level. Level 0 is the highest resolution. downres "false" (default) or "true", specifies whether the given blocks should be down-sampled to lower resolution. If "true", scale must be "0" or absent. compression Specifies compression format of block data: default and only option currently is "blocks" (native DVID label blocks). throttle If "true", makes sure only N compute-intense operation (all API calls that can be throttled) are handled. If the server can't initiate the API call right away, a 503 (Service Unavailable) status code is returned. GET /node///maxlabel GET returns the maximum label for the version of data in JSON form: { "maxlabel":