Apache Hadoop 3.4.1 – WebHDFS REST API (original) (raw)

Document Conventions

Monospaced Used for commands, HTTP request and responses and code blocks.
User entered values.
[Monospaced] Optional values. When the value is not specified, the default value is used.
Italics Important phrases and words.

Introduction

The HTTP REST API supports the complete FileSystem/FileContext interface for HDFS. The operations and the corresponding FileSystem/FileContext methods are shown in the next section. The Section HTTP Query Parameter Dictionary specifies the parameter details such as the defaults and the valid values.

Operations

FileSystem URIs vs HTTP URLs

The FileSystem scheme of WebHDFS is “webhdfs://”. A WebHDFS FileSystem URI has the following format.

webhdfs://:/

The above WebHDFS URI corresponds to the below HDFS URI.

hdfs://:/

In the REST API, the prefix “/webhdfs/v1” is inserted in the path and a query is appended at the end. Therefore, the corresponding HTTP URL has the following format.

http://:/webhdfs/v1/?op=...

Note that if WebHDFS is secured with SSL, then the scheme should be “swebhdfs://”.

swebhdfs://:/

See also: SSL Configurations for SWebHDFS

HDFS Configuration Options

Below are the HDFS configuration options for WebHDFS.

Property Name Description
dfs.web.authentication.kerberos.principal The HTTP Kerberos principal used by Hadoop-Auth in the HTTP endpoint. The HTTP Kerberos principal MUST start with ‘HTTP/’ per Kerberos HTTP SPNEGO specification. A value of “*” will use all HTTP principals found in the keytab.
dfs.web.authentication.kerberos.keytab The Kerberos keytab file with the credentials for the HTTP Kerberos principal used by Hadoop-Auth in the HTTP endpoint.
dfs.webhdfs.socket.connect-timeout How long to wait for a connection to be established before failing. Specified as a time duration, ie numerical value followed by a units symbol, eg 2m for two minutes. Defaults to 60s.
dfs.webhdfs.socket.read-timeout How long to wait for data to arrive before failing. Defaults to 60s.

Authentication

When security is off, the authenticated user is the username specified in the user.name query parameter. If the user.name parameter is not set, the server may either set the authenticated user to a default web user, if there is any, or return an error response.

When security is on, authentication is performed by either Hadoop delegation token or Kerberos SPNEGO. If a token is set in the delegation query parameter, the authenticated user is the user encoded in the token. If the delegation parameter is not set, the user is authenticated by Kerberos SPNEGO.

Below are examples using the curl command tool.

  1. Authentication when security is off:
    curl -i "http://:/webhdfs/v1/?[user.name=&]op=..."
  2. Authentication using Kerberos SPNEGO when security is on:
    curl -i --negotiate -u : "http://:/webhdfs/v1/?op=..."
  3. Authentication using Hadoop delegation token when security is on:
    curl -i "http://:/webhdfs/v1/?delegation=&op=..."

See also: Authentication for Hadoop HTTP web-consoles

Additionally, WebHDFS supports OAuth2 on the client side. The Namenode and Datanodes do not currently support clients using OAuth2 but other backends that implement the WebHDFS REST interface may.

WebHDFS supports two type of OAuth2 code grants (user-provided refresh and access token or user provided credential) by default and provides a pluggable mechanism for implementing other OAuth2 authentications per the OAuth2 RFC, or custom authentications. When using either of the provided code grant mechanisms, the WebHDFS client will refresh the access token as necessary.

OAuth2 should only be enabled for clients not running with Kerberos SPENGO.

OAuth2 code grant mechanism Description Value of dfs.webhdfs.oauth2.access.token.provider that implements code grant
Authorization Code Grant The user provides an initial access token and refresh token, which are then used to authenticate WebHDFS requests and obtain replacement access tokens, respectively. org.apache.hadoop.hdfs.web.oauth2.ConfRefreshTokenBasedAccessTokenProvider
Client Credentials Grant The user provides a credential which is used to obtain access tokens, which are then used to authenticate WebHDFS requests. org.apache.hadoop.hdfs.web.oauth2.ConfCredentialBasedAccessTokenProvider

The following properties control OAuth2 authentication.

OAuth2 related property Description
dfs.webhdfs.oauth2.enabled Boolean to enable/disable OAuth2 authentication
dfs.webhdfs.oauth2.access.token.provider Class name of an implementation of org.apache.hadoop.hdfs.web.oauth.AccessTokenProvider. Two are provided with the code, as described above, or the user may specify a user-provided implementation. The default value for this configuration key is the ConfCredentialBasedAccessTokenProvider implementation.
dfs.webhdfs.oauth2.client.id Client id used to obtain access token with either credential or refresh token
dfs.webhdfs.oauth2.refresh.url URL against which to post for obtaining bearer token with either credential or refresh token
dfs.webhdfs.oauth2.access.token (required if using ConfRefreshTokenBasedAccessTokenProvider) Initial access token with which to authenticate
dfs.webhdfs.oauth2.refresh.token (required if using ConfRefreshTokenBasedAccessTokenProvider) Initial refresh token to use to obtain new access tokens
dfs.webhdfs.oauth2.refresh.token.expires.ms.since.epoch (required if using ConfRefreshTokenBasedAccessTokenProvider) Access token expiration measured in milliseconds since Jan 1, 1970. Note this is a different value than provided by OAuth providers and has been munged as described in interface to be suitable for a client application
dfs.webhdfs.oauth2.credential (required if using ConfCredentialBasedAccessTokenProvider). Credential used to obtain initial and subsequent access tokens.

SSL Configurations for SWebHDFS

To use SWebHDFS FileSystem (i.e. using the swebhdfs protocol), a SSL configuration file needs to be specified on the client side. This must specify 3 parameters:

SSL property Description
ssl.client.truststore.location The local-filesystem location of the trust-store file, containing the certificate for the NameNode.
ssl.client.truststore.type (Optional) The format of the trust-store file.
ssl.client.truststore.password (Optional) Password for the trust-store file.

The following is an example SSL configuration file (ssl-client.xml):

ssl.client.truststore.location /work/keystore.jks Truststore to be used by clients. Must be specified. ssl.client.truststore.password changeme Optional. Default value is "". ssl.client.truststore.type jks Optional. Default value is "jks".

The SSL configuration file must be in the class-path of the client program and the filename needs to be specified in core-site.xml:

hadoop.ssl.client.conf ssl-client.xml Resource file from which ssl client keystore information will be extracted. This file is looked up in the classpath, typically it should be in Hadoop conf/ directory. Default value is "ssl-client.xml".

Proxy Users

When the proxy user feature is enabled, a proxy user P may submit a request on behalf of another user U. The username of U must be specified in the doas query parameter unless a delegation token is presented in authentication. In such case, the information of both users P and U must be encoded in the delegation token.

  1. A proxy request when security is off:
    curl -i "http://:/webhdfs/v1/?[user.name=&]doas=&op=..."
  2. A proxy request using Kerberos SPNEGO when security is on:
    curl -i --negotiate -u : "http://:/webhdfs/v1/?doas=&op=..."
  3. A proxy request using Hadoop delegation token when security is on:
    curl -i "http://:/webhdfs/v1/?delegation=&op=..."

Cross-Site Request Forgery Prevention

WebHDFS supports an optional, configurable mechanism for cross-site request forgery (CSRF) prevention. When enabled, WebHDFS HTTP requests to the NameNode or DataNode must include a custom HTTP header. Configuration properties allow adjusting which specific HTTP methods are protected and the name of the HTTP header. The value sent in the header is not relevant. Only the presence of a header by that name is required.

Enabling CSRF prevention also sets up the WebHdfsFileSystem class to send the required header. This ensures that CLI commands like hdfs dfs and hadoop distcp continue to work correctly when used with webhdfs: URIs.

Enabling CSRF prevention also sets up the NameNode web UI to send the required header. After enabling CSRF prevention and restarting the NameNode, existing users of the NameNode web UI need to refresh the browser to reload the page and find the new configuration.

The following properties control CSRF prevention.

Property Description Default Value
dfs.webhdfs.rest-csrf.enabled If true, then enables WebHDFS protection against cross-site request forgery (CSRF). The WebHDFS client also uses this property to determine whether or not it needs to send the custom CSRF prevention header in its HTTP requests. false
dfs.webhdfs.rest-csrf.custom-header The name of a custom header that HTTP requests must send when protection against cross-site request forgery (CSRF) is enabled for WebHDFS by setting dfs.webhdfs.rest-csrf.enabled to true. The WebHDFS client also uses this property to determine whether or not it needs to send the custom CSRF prevention header in its HTTP requests. X-XSRF-HEADER
dfs.webhdfs.rest-csrf.methods-to-ignore A comma-separated list of HTTP methods that do not require HTTP requests to include a custom header when protection against cross-site request forgery (CSRF) is enabled for WebHDFS by setting dfs.webhdfs.rest-csrf.enabled to true. The WebHDFS client also uses this property to determine whether or not it needs to send the custom CSRF prevention header in its HTTP requests. GET,OPTIONS,HEAD,TRACE
dfs.webhdfs.rest-csrf.browser-useragents-regex A comma-separated list of regular expressions used to match against an HTTP request’s User-Agent header when protection against cross-site request forgery (CSRF) is enabled for WebHDFS by setting dfs.webhdfs.reset-csrf.enabled to true. If the incoming User-Agent matches any of these regular expressions, then the request is considered to be sent by a browser, and therefore CSRF prevention is enforced. If the request’s User-Agent does not match any of these regular expressions, then the request is considered to be sent by something other than a browser, such as scripted automation. In this case, CSRF is not a potential attack vector, so the prevention is not enforced. This helps achieve backwards-compatibility with existing automation that has not been updated to send the CSRF prevention header. ^Mozilla.*,^Opera.*
dfs.datanode.httpserver.filter.handlers Comma separated list of Netty servlet-style filter handlers to inject into the Datanode WebHDFS I/O path org.apache.hadoop.hdfs.server.datanode.web.RestCsrfPreventionFilterHandler

The following is an example curl call that uses the -H option to include the custom header in the request.

curl -i -L -X PUT -H 'X-XSRF-HEADER: ""' 'http://<HOST>:<PORT>/webhdfs/v1/<PATH>?op=CREATE'

WebHDFS Retry Policy

WebHDFS supports an optional, configurable retry policy for resilient copy of large files that could timeout, or copy file between HA clusters that could failover during the copy.

The following properties control WebHDFS retry and failover policy.

Property Description Default Value
dfs.http.client.retry.policy.enabled If “true”, enable the retry policy of WebHDFS client. If “false”, retry policy is turned off. false
dfs.http.client.retry.policy.spec Specify a policy of multiple linear random retry for WebHDFS client, e.g. given pairs of number of retries and sleep time (n0, t0), (n1, t1), …, the first n0 retries sleep t0 milliseconds on average, the following n1 retries sleep t1 milliseconds on average, and so on. 10000,6,60000,10
dfs.http.client.failover.max.attempts Specify the max number of failover attempts for WebHDFS client in case of network exception. 15
dfs.http.client.retry.max.attempts Specify the max number of retry attempts for WebHDFS client, if the difference between retried attempts and failovered attempts is larger than the max number of retry attempts, there will be no more retries. 10
dfs.http.client.failover.sleep.base.millis Specify the base amount of time in milliseconds upon which the exponentially increased sleep time between retries or failovers is calculated for WebHDFS client. 500
dfs.http.client.failover.sleep.max.millis Specify the upper bound of sleep time in milliseconds between retries or failovers for WebHDFS client. 15000

WebHDFS Request Filtering

One may control directionality of data in the WebHDFS protocol allowing only writing data from insecure networks. To enable, one must ensure dfs.datanode.httpserver.filter.handlers includes org.apache.hadoop.hdfs.server.datanode.web.HostRestrictingAuthorizationFilterHandler. Configuration of the HostRestrictingAuthorizationFilter is controlled via the following properties.

Property Description Default Value
dfs.datanode.httpserver.filter.handlers Comma separated list of Netty servlet-style filter handlers to inject into the Datanode WebHDFS I/O path org.apache.hadoop.hdfs.server.datanode.web.RestCsrfPreventionFilterHandler
dfs.web.authentication.host.allow.rules Rules allowing users to read files in the format of user,network/bits,path glob newline or |-separated. Use * for a wildcard of all users or network/bits. nothing - defaults to no one may read via WebHDFS

File and Directory Operations

Create and Write to a File

Usually the request is redirected to a datanode where the file data is to be written.
HTTP/1.1 307 TEMPORARY_REDIRECT
Location: http://:/webhdfs/v1/?op=CREATE...
Content-Length: 0
However, if you do not want to be automatically redirected, you can set the noredirect flag.
HTTP/1.1 200 OK
Content-Type: application/json
{"Location":"http://:/webhdfs/v1/?op=CREATE..."}

Note that the reason of having two-step create/append is for preventing clients to send out data before the redirect. This issue is addressed by the “Expect: 100-continue” header in HTTP/1.1; see RFC 2616, Section 8.2.3. Unfortunately, there are software library bugs (e.g. Jetty 6 HTTP server and Java 6 HTTP client), which do not correctly implement “Expect: 100-continue”. The two-step create/append is a temporary workaround for the software library bugs.

See also: overwrite, blocksize, replication, permission, buffersize, FileSystem.create

Append to a File

However, if you do not want to be automatically redirected, you can set the noredirect flag.

HTTP/1.1 200 OK
Content-Type: application/json
{"Location":"http://<DATANODE>:<PORT>/webhdfs/v1/<PATH>?op=APPEND..."}

See the note in the previous section for the description of why this operation requires two steps.

See also: buffersize, FileSystem.append

Concat File(s)

See also: sources, FileSystem.concat

Open and Read a File

Usually the request is redirected to a datanode where the file data can be read:
HTTP/1.1 307 TEMPORARY_REDIRECT
Location: http://:/webhdfs/v1/?op=OPEN...
Content-Length: 0
However if you do not want to be automatically redirected, you can set the noredirect flag.
HTTP/1.1 200 OK
Content-Type: application/json
{"Location":"http://:/webhdfs/v1/?op=OPEN..."}
The client follows the redirect to the datanode and receives the file data:
HTTP/1.1 200 OK
Content-Type: application/octet-stream
Content-Length: 22
Hello, webhdfs user!

See also: offset, length, buffersize, FileSystem.open

Make a Directory

See also: permission, FileSystem.mkdirs

The client receives a response with zero content length:
HTTP/1.1 200 OK
Content-Length: 0

See also: destination, createParent, FileSystem.createSymlink

Rename a File/Directory

See also: destination, FileSystem.rename

Delete a File/Directory

The client receives a response with a boolean JSON object:
HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked
{"boolean": true}

See also: recursive, FileSystem.delete

Truncate a File

See also: newlength, FileSystem.truncate

Status of a File/Directory

See also: FileSystem.getFileStatus

List a Directory

See also: FileSystem.listStatus

List a File

See also: FileSystem.listStatus

Iteratively List a Directory

If remainingEntries is non-zero, there are additional entries in the directory. To query the next batch, set the startAfter parameter to the pathSuffix of the last item returned in the current batch. For example:

curl -i  "http://<HOST>:<PORT>/webhdfs/v1/<PATH>?op=LISTSTATUS_BATCH&startAfter=bazfile"

Which will return the next batch of directory entries:

HTTP/1.1 200 OK
Cache-Control: no-cache
Expires: Thu, 08 Sep 2016 03:43:20 GMT
Date: Thu, 08 Sep 2016 03:43:20 GMT
Pragma: no-cache
Expires: Thu, 08 Sep 2016 03:43:20 GMT
Date: Thu, 08 Sep 2016 03:43:20 GMT
Pragma: no-cache
Content-Type: application/json
X-FRAME-OPTIONS: SAMEORIGIN
Transfer-Encoding: chunked
Server: Jetty(6.1.26)

{
    "DirectoryListing": {
        "partialListing": {
            "FileStatuses": {
                "FileStatus": [
                    {
                        "accessTime": 0,
                        "blockSize": 0,
                        "childrenNum": 0,
                        "fileId": 16386,
                        "group": "supergroup",
                        "length": 0,
                        "modificationTime": 1473305878951,
                        "owner": "andrew",
                        "pathSuffix": "foodir",
                        "permission": "755",
                        "replication": 0,
                        "storagePolicy": 0,
                        "type": "DIRECTORY"
                    },
                    {
                        "accessTime": 1473305902864,
                        "blockSize": 1024,
                        "childrenNum": 0,
                        "fileId": 16389,
                        "group": "supergroup",
                        "length": 0,
                        "modificationTime": 1473305902878,
                        "owner": "andrew",
                        "pathSuffix": "quxfile",
                        "permission": "644",
                        "replication": 3,
                        "storagePolicy": 0,
                        "type": "FILE"
                    }
                ]
            }
        },
        "remainingEntries": 0
    }
}

Batch size is controlled by the dfs.ls.limit option on the NameNode.

See also: FileSystem.listStatusIterator

Other File System Operations

Get Content Summary of a Directory

See also: FileSystem.getContentSummary

Get Quota Usage of a Directory

See also: FileSystem.getQuotaUsage

Set Quota

The client receives a response with zero content length:
HTTP/1.1 200 OK
Content-Length: 0

See also: FileSystem.setQuota

Set Quota By Storage Type

The client receives a response with zero content length:
HTTP/1.1 200 OK
Content-Length: 0

See also: FileSystem.setQuotaByStorageType

Get File Checksum

See also: FileSystem.getFileChecksum

Get Home Directory

See also: FileSystem.getHomeDirectory

Get Trash Root

See also: FileSystem.getTrashRoot

For more details about trash root in an encrypted zone, please refer to Transparent Encryption Guide.

Set Permission

The client receives a response with zero content length:
HTTP/1.1 200 OK
Content-Length: 0

See also: permission, FileSystem.setPermission

Set Owner

The client receives a response with zero content length:
HTTP/1.1 200 OK
Content-Length: 0

See also: owner, group, FileSystem.setOwner

Set Replication Factor

The client receives a response with a boolean JSON object:
HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked
{"boolean": true}

See also: replication, FileSystem.setReplication

Set Access or Modification Time

The client receives a response with zero content length:
HTTP/1.1 200 OK
Content-Length: 0

See also: modificationtime, accesstime, FileSystem.setTimes

Modify ACL Entries

The client receives a response with zero content length:
HTTP/1.1 200 OK
Content-Length: 0

See also: FileSystem.modifyAclEntries

Remove ACL Entries

The client receives a response with zero content length:
HTTP/1.1 200 OK
Content-Length: 0

See also: FileSystem.removeAclEntries

Remove Default ACL

See also: FileSystem.removeDefaultAcl

Remove ACL

See also: FileSystem.removeAcl

Set ACL

The client receives a response with zero content length:
HTTP/1.1 200 OK
Content-Length: 0

See also: FileSystem.setAcl

Get ACL Status

See also: FileSystem.getAclStatus

Check access

The client receives a response with zero content length:
HTTP/1.1 200 OK
Content-Length: 0

See also: FileSystem.access

Get Server Defaults

The client receives a response with a ServerDefaults JSON object:

HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked

{
    "FsServerDefaults": {
        "replication": 3,
        "encryptDataTransfer": "false",
        "defaultStoragePolicyId":7,
        "writePacketSize": 65536,
        "fileBufferSize": 4096,
        "checksumType": 2,
        "trashInterval": 10080,
        "keyProviderUri": "",
        "blockSize": 134217728,
        "bytesPerChecksum": 512
    }
}

See also: FileSystem.getServerDefaults

See also: FileSystem.getLinkTarget

The client receives a response with a FileStatus JSON object:

HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked

{
    "FileStatus": {
        "accessTime": 0,
        "blockSize": 0,
        "childrenNum":0,
        "fileId": 16388,
        "group": "supergroup",
        "length": 0,
        "modificationTime": 1681916788427,
        "owner": "hadoop",
        "pathSuffix": "",
        "permission": "777",
        "replication": 0,
        "storagePolicy": 0,
        "symlink": "/webHdfsTest/file",
        "type": "SYMLINK"
    }
}

See also: FileSystem.getFileLinkInfo

Get EC Policies

The client receives a response with a ECPolicies JSON object:

HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked

{
  "ErasureCodingPolicies": {
    "ErasureCodingPolicyInfo": [
      {
        "state": "ENABLED",
        "policy": {
          "name": "RS-6-3-1024k",
          "schema": {
            "codecName": "rs",
            "numDataUnits": 6,
            "numParityUnits": 3,
            "extraOptions": {}
          },
          "cellSize": 1048576,
          "id": 1,
          "replicationPolicy": false,
          "codecName": "rs",
          "numDataUnits": 6,
          "numParityUnits": 3,
          "systemPolicy": true
        }
      }
    ]
  }
}

Get Status

The client receives a response with a FsStatus JSON object:

HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked

{
    "FsStatus": {
        "used": 29229154304,
        "remaining": 292893392896,
        "capacity":322122547200
    }
}

See also: FileSystem.getStatus

Get EC Codecs

The client receives a response with a ECCodecs JSON object:

HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked

{
    "ErasureCodeCodecs": {
        "rs": "rs_native, rs_java",
        "rs-legacy": "rs-legacy_java",
        "xor":"xor_native, xor_java"
    }
}

Get Trash Roots

The client receives a response with a Paths JSON object:

HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked

{
    "Paths": [{
        "blocksize": 0,
        "owner": "hadoop",
        "path": "/user/user0/.Trash",
        "length": 0,
        "permission": "755",
        "modification_time": 1693050205747,
        "isdir": true,
        "block_replication": 0,
        "access_time": 0,
        "group": "supergroup"
     }, {
        "blocksize": 0,
        "owner": "hadoop",
        "path": "/user/user1/.Trash",
        "length": 0,
        "permission": "755",
        "modification_time": 1693049382962,
        "isdir": true,
        "block_replication": 0,
        "access_time": 0,
        "group": "supergroup"
     }]
}

See also: FileSystem.getTrashRoots

Storage Policy Operations

Get all Storage Policies

See also: FileSystem.getAllStoragePolicies

Set Storage Policy

The client receives a response with zero content length:
HTTP/1.1 200 OK
Content-Length: 0

See also: FileSystem.setStoragePolicy

Unset Storage Policy

See also: FileSystem.unsetStoragePolicy

Get Storage Policy

See also: FileSystem.getStoragePolicy

Satisfy Storage Policy

See also: ArchivalStorage.satisfyStoragePolicy

Get File Block Locations

See also: offset, length, FileSystem.getFileBlockLocations

Extended Attributes(XAttrs) Operations

Set XAttr

The client receives a response with zero content length:
HTTP/1.1 200 OK
Content-Length: 0

See also: FileSystem.setXAttr

Remove XAttr

The client receives a response with zero content length:
HTTP/1.1 200 OK
Content-Length: 0

See also: FileSystem.removeXAttr

Get an XAttr

The client receives a response with a XAttrs JSON object:
HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked
{
"XAttrs": [
{
"name":"XATTRNAME",
"value":"XATTRVALUE"
}
]
}

See also: FileSystem.getXAttr

Get multiple XAttrs

The client receives a response with a XAttrs JSON object:
HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked
{
"XAttrs": [
{
"name":"XATTRNAME1",
"value":"XATTRVALUE1"
},
{
"name":"XATTRNAME2",
"value":"XATTRVALUE2"
}
]
}

See also: FileSystem.getXAttrs

Get all XAttrs

The client receives a response with a XAttrs JSON object:
HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked
{
"XAttrs": [
{
"name":"XATTRNAME1",
"value":"XATTRVALUE1"
},
{
"name":"XATTRNAME2",
"value":"XATTRVALUE2"
},
{
"name":"XATTRNAME3",
"value":"XATTRVALUE3"
}
]
}

See also: FileSystem.getXAttrs

List all XAttrs

See also: FileSystem.listXAttrs

Erasure Coding Operations

Enable EC Policy

The client receives a response with zero content length:
HTTP/1.1 200 OK
Content-Length: 0

See also: HDFSErasureCoding.enablePolicy

Disable EC Policy

The client receives a response with zero content length:
HTTP/1.1 200 OK
Content-Length: 0

See also: HDFSErasureCoding.disablePolicy

Set EC Policy

The client receives a response with zero content length:
HTTP/1.1 200 OK
Content-Length: 0

See also: HDFSErasureCoding.setErasureCodingPolicy

Get EC Policy

The client receives a response with a ECPolicy JSON object:

{
    "name": "RS-10-4-1024k",
    "schema":
    {
    "codecName": "rs",
    "numDataUnits": 10,
    "numParityUnits": 4,
    "extraOptions": {}
    }
    "cellSize": 1048576,
    "id":5,
    "codecname":"rs",
    "numDataUnits": 10,
    "numParityUnits": 4,
    "replicationpolicy":false,
    "systemPolicy":true

}

See also: HDFSErasureCoding.getErasureCodingPolicy

Unset EC Policy

The client receives a response with zero content length:
HTTP/1.1 200 OK
Content-Length: 0

See also: HDFSErasureCoding.unsetErasureCodingPolicy

Snapshot Operations

Allow Snapshot

Disallow Snapshot

Create Snapshot

See also: FileSystem.createSnapshot

Delete Snapshot

See also: FileSystem.deleteSnapshot

Rename Snapshot

The client receives a response with zero content length:
HTTP/1.1 200 OK
Content-Length: 0

See also: FileSystem.renameSnapshot

Get Snapshot Diff

The client receives a response with a SnapshotDiffReport JSON object:
HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked
{"SnapshotDiffReport":{"diffList":[],"fromSnapshot":"s3","snapshotRoot":"/foo","toSnapshot":"s4"}}

Get Snapshot Diff Iteratively

If snapshotdiffstartpath and snapshotdiffindex are not given, "" (empty string) and -1 are used respectively implying the first iteration.
The client receives a response with a SnapshotDiffReportListing JSON object. The value of lastPath and lastIndex must be specified as the value of snapshotdiffstartpath and snapshotdiffindex respectively on next iteration.
HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked
{"SnapshotDiffReportListing":{"createList":[],"deleteList":[],"isFromEarlier":true,"lastIndex":-1,"lastPath":"","modifyList":[]}}

Get Snapshottable Directory List

Get Snapshot List

Delegation Token Operations

Get Delegation Token

The client receives a response with a Token JSON object:
HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked
{
"Token":
{
"urlString": "JQAIaG9y..."
}
}

See also: renewer, FileSystem.getDelegationToken, kind, service

Renew Delegation Token

See also: token, DelegationTokenAuthenticator.renewDelegationToken

Cancel Delegation Token

See also: token, DelegationTokenAuthenticator.cancelDelegationToken

Error Responses

When an operation fails, the server may throw an exception. The JSON schema of error responses is defined in RemoteException JSON Schema. The table below shows the mapping from exceptions to HTTP response codes.

HTTP Response Codes

Exceptions HTTP Response Codes
IllegalArgumentException 400 Bad Request
UnsupportedOperationException 400 Bad Request
SecurityException 401 Unauthorized
IOException 403 Forbidden
FileNotFoundException 404 Not Found
RuntimeException 500 Internal Server Error

Below are examples of exception responses.

Illegal Argument Exception

HTTP/1.1 400 Bad Request Content-Type: application/json Transfer-Encoding: chunked

{ "RemoteException": { "exception" : "IllegalArgumentException", "javaClassName": "java.lang.IllegalArgumentException", "message" : "Invalid value for webhdfs parameter "permission": ..." } }

Security Exception

HTTP/1.1 401 Unauthorized Content-Type: application/json Transfer-Encoding: chunked

{ "RemoteException": { "exception" : "SecurityException", "javaClassName": "java.lang.SecurityException", "message" : "Failed to obtain user group information: ..." } }

Access Control Exception

HTTP/1.1 403 Forbidden Content-Type: application/json Transfer-Encoding: chunked

{ "RemoteException": { "exception" : "AccessControlException", "javaClassName": "org.apache.hadoop.security.AccessControlException", "message" : "Permission denied: ..." } }

File Not Found Exception

HTTP/1.1 404 Not Found Content-Type: application/json Transfer-Encoding: chunked

{ "RemoteException": { "exception" : "FileNotFoundException", "javaClassName": "java.io.FileNotFoundException", "message" : "File does not exist: /foo/a.patch" } }

JSON Schemas

All operations, except for OPEN, either return a zero-length response or a JSON response. For OPEN, the response is an octet-stream. The JSON schemas are shown below. See draft-zyp-json-schema-03 for the syntax definitions of the JSON schemas.

Note that the default value of additionalProperties is an empty schema which allows any value for additional properties. Therefore, all WebHDFS JSON responses allow any additional property. However, if additional properties are included in the responses, they are considered as optional properties in order to maintain compatibility.

ACL Status JSON Schema

{ "name" : "AclStatus", "properties": { "AclStatus": { "type" : "object", "properties": { "entries": { "type": "array", "items": { "description": "ACL entry.", "type": "string" } }, "group": { "description": "The group owner.", "type" : "string", "required" : true }, "owner": { "description": "The user who is the owner.", "type" : "string", "required" : true }, "stickyBit": { "description": "True if the sticky bit is on.", "type" : "boolean", "required" : true } } } } }

XAttrs JSON Schema

{ "name" : "XAttrs", "properties": { "XAttrs": { "type" : "array", "items": { "type" : "object", "properties": { "name": { "description": "XAttr name.", "type" : "string", "required" : true }, "value": { "description": "XAttr value.", "type" : "string" } } } } } }

XAttrNames JSON Schema

{ "name" : "XAttrNames", "properties": { "XAttrNames": { "description": "XAttr names.", "type" : "string", "required" : true } } }

Boolean JSON Schema

{ "name" : "boolean", "properties": { "boolean": { "description": "A boolean value", "type" : "boolean", "required" : true } } }

See also: MKDIRS, RENAME, DELETE, SETREPLICATION

ContentSummary JSON Schema

{ "name" : "ContentSummary", "properties": { "ContentSummary": { "type" : "object", "properties": { "directoryCount": { "description": "The number of directories.", "type" : "integer", "required" : true }, "fileCount": { "description": "The number of files.", "type" : "integer", "required" : true }, "length": { "description": "The number of bytes used by the content.", "type" : "integer", "required" : true }, "quota": { "description": "The namespace quota of this directory.", "type" : "integer", "required" : true }, "spaceConsumed": { "description": "The disk space consumed by the content.", "type" : "integer", "required" : true }, "spaceQuota": { "description": "The disk space quota.", "type" : "integer", "required" : true }, "typeQuota": { "type" : "object", "properties": { "ARCHIVE": { "type" : "object", "properties": { "consumed": { "description": "The storage type space consumed.", "type" : "integer", "required" : true }, "quota": { "description": "The storage type quota.", "type" : "integer", "required" : true } } }, "DISK": { "type" : "object", "properties": { "consumed": { "description": "The storage type space consumed.", "type" : "integer", "required" : true }, "quota": { "description": "The storage type quota.", "type" : "integer", "required" : true } } }, "SSD": { "type" : "object", "properties": { "consumed": { "description": "The storage type space consumed.", "type" : "integer", "required" : true }, "quota": { "description": "The storage type quota.", "type" : "integer", "required" : true } } } } } } } } }

See also: GETCONTENTSUMMARY

QuotaUsage JSON Schema

{ "name" : "QuotaUsage", "properties": { "QuotaUsage": { "type" : "object", "properties": { "fileAndDirectoryCount": { "description": "The number of files and directories.", "type" : "integer", "required" : true }, "quota": { "description": "The namespace quota of this directory.", "type" : "integer", "required" : true }, "spaceConsumed": { "description": "The disk space consumed by the content.", "type" : "integer", "required" : true }, "spaceQuota": { "description": "The disk space quota.", "type" : "integer", "required" : true }, "typeQuota": { "type" : "object", "properties": { "ARCHIVE": { "type" : "object", "properties": { "consumed": { "description": "The storage type space consumed.", "type" : "integer", "required" : true }, "quota": { "description": "The storage type quota.", "type" : "integer", "required" : true } } }, "DISK": { "type" : "object", "properties": { "consumed": { "description": "The storage type space consumed.", "type" : "integer", "required" : true }, "quota": { "description": "The storage type quota.", "type" : "integer", "required" : true } } }, "SSD": { "type" : "object", "properties": { "consumed": { "description": "The storage type space consumed.", "type" : "integer", "required" : true }, "quota": { "description": "The storage type quota.", "type" : "integer", "required" : true } } } } } } } } }

See also: GETQUOTAUSAGE

FileChecksum JSON Schema

{ "name" : "FileChecksum", "properties": { "FileChecksum": { "type" : "object", "properties": { "algorithm": { "description": "The name of the checksum algorithm.", "type" : "string", "required" : true }, "bytes": { "description": "The byte sequence of the checksum in hexadecimal.", "type" : "string", "required" : true }, "length": { "description": "The length of the bytes (not the length of the string).", "type" : "integer", "required" : true } } } } }

FileStatus JSON Schema

{ "name" : "FileStatus", "properties": { "FileStatus": fileStatusProperties //See FileStatus Properties } }

See also: FileStatus Properties, GETFILESTATUS, FileStatus

FileStatus Properties

JavaScript syntax is used to define fileStatusProperties so that it can be referred in both FileStatus and FileStatuses JSON schemas.

var fileStatusProperties = { "type" : "object", "properties": { "accessTime": { "description": "The access time.", "type" : "integer", "required" : true }, "blockSize": { "description": "The block size of a file.", "type" : "integer", "required" : true }, "group": { "description": "The group owner.", "type" : "string", "required" : true }, "length": { "description": "The number of bytes in a file.", "type" : "integer", "required" : true }, "modificationTime": { "description": "The modification time.", "type" : "integer", "required" : true }, "owner": { "description": "The user who is the owner.", "type" : "string", "required" : true }, "pathSuffix": { "description": "The path suffix.", "type" : "string", "required" : true }, "permission": { "description": "The permission represented as a octal string.", "type" : "string", "required" : true }, "replication": { "description": "The number of replication of a file.", "type" : "integer", "required" : true }, "symlink": //an optional property { "description": "The link target of a symlink.", "type" : "string" }, "type": { "description": "The type of the path object.", "enum" : ["FILE", "DIRECTORY", "SYMLINK"], "required" : true }, "aclBit": { "description": "Has ACLs set or not.", "type" : "boolean", }, "encBit": { "description": "Is Encrypted or not.", "type" : "boolean", }, "ecBit": { "description": "Is ErasureCoded or not.", "type" : "boolean", }, "ecPolicy": { "description": "The namenode of ErasureCodePolicy.", "type" : "String", } } };

FileStatuses JSON Schema

A FileStatuses JSON object represents an array of FileStatus JSON objects.

{ "name" : "FileStatuses", "properties": { "FileStatuses": { "type" : "object", "properties": { "FileStatus": { "description": "An array of FileStatus", "type" : "array", "items" : fileStatusProperties //See FileStatus Properties } } } } }

See also: FileStatus Properties, LISTSTATUS, FileStatus

DirectoryListing JSON Schema

A DirectoryListing JSON object represents a batch of directory entries while iteratively listing a directory. It contains a FileStatuses JSON object as well as iteration information.

{ "name" : "DirectoryListing", "properties": { "DirectoryListing": { "type" : "object", "properties": { "partialListing": { "description": "A partial directory listing", "type" : "object", // A FileStatuses object "required" : true }, "remainingEntries": { "description": "Number of remaining entries", "type" : "integer", "required" : true } } } }

}

See also: FileStatuses JSON Schema, LISTSTATUS_BATCH, FileStatus

Long JSON Schema

{ "name" : "long", "properties": { "long": { "description": "A long integer value", "type" : "integer", "required" : true } } }

See also: RENEWDELEGATIONTOKEN,

Path JSON Schema

{ "name" : "Path", "properties": { "Path": { "description": "The string representation a Path.", "type" : "string", "required" : true } } }

See also: GETHOMEDIRECTORY, Path

RemoteException JSON Schema

{ "name" : "RemoteException", "properties": { "RemoteException": { "type" : "object", "properties": { "exception": { "description": "Name of the exception", "type" : "string", "required" : true }, "message": { "description": "Exception message", "type" : "string", "required" : true }, "javaClassName": //an optional property { "description": "Java class name of the exception", "type" : "string" } } } } }

See also: Error Responses

Token JSON Schema

{ "name" : "Token", "properties": { "Token": tokenProperties //See Token Properties } }

See also: Token Properties, GETDELEGATIONTOKEN, the note in Delegation.

Token Properties

JavaScript syntax is used to define tokenProperties so that it can be referred in Token JSON schema.

var tokenProperties = { "type" : "object", "properties": { "urlString": { "description": "A delegation token encoded as a URL safe string.", "type" : "string", "required" : true } } }

See also: Token Properties, the note in Delegation.

BlockStoragePolicy JSON Schema

{ "name" : "BlockStoragePolicy", "properties": { "BlockStoragePolicy": blockStoragePolicyProperties //See BlockStoragePolicy Properties } }

See also: BlockStoragePolicy Properties, GETSTORAGEPOLICY

BlockStoragePolicy Properties

JavaScript syntax is used to define blockStoragePolicyProperties so that it can be referred in both BlockStoragePolicy and BlockStoragePolicies JSON schemas.

var blockStoragePolicyProperties = { "type" : "object", "properties": { "id": { "description": "Policy ID.", "type" : "integer", "required" : true }, "name": { "description": "Policy name.", "type" : "string", "required" : true }, "storageTypes": { "description": "An array of storage types for block placement.", "type" : "array", "required" : true "items" : { "type": "string" } }, "replicationFallbacks": { "description": "An array of fallback storage types for replication.", "type" : "array", "required" : true "items" : { "type": "string" } }, "creationFallbacks": { "description": "An array of fallback storage types for file creation.", "type" : "array", "required" : true "items" : { "type": "string" } }, "copyOnCreateFile": { "description": "If set then the policy cannot be changed after file creation.", "type" : "boolean", "required" : true } } };

ECPolicy JSON Schema

{ "name": "RS-10-4-1024k", schema { "codecName": "rs", "numDataUnits": 10, "numParityUnits": 4, "extraOptions": {} } "cellSize": 1048576, "id":5, "codecname":"rs", "numDataUnits": 10, "numParityUnits": 4, "replicationpolicy":false, "systemPolicy":true }

BlockStoragePolicies JSON Schema

A BlockStoragePolicies JSON object represents an array of BlockStoragePolicy JSON objects.

{ "name" : "BlockStoragePolicies", "properties": { "BlockStoragePolicies": { "type" : "object", "properties": { "BlockStoragePolicy": { "description": "An array of BlockStoragePolicy", "type" : "array", "items" : blockStoragePolicyProperties //See BlockStoragePolicy Properties } } } } }

SnapshotDiffReport JSON Schema

{ "name": "SnapshotDiffReport", "type": "object", "properties": { "SnapshotDiffReport": { "type" : "object", "properties" : { "diffList": { "description": "An array of DiffReportEntry", "type" : "array", "items" : diffReportEntries, "required" : true }, "fromSnapshot": { "description": "Source snapshot", "type" : "string", "required" : true }, "snapshotRoot": { "description" : "String representation of snapshot root path", "type" : "string", "required" : true }, "toSnapshot": { "description" : "Destination snapshot", "type" : "string", "required" : true } } } } }

DiffReport Entries

JavaScript syntax is used to define diffReportEntries so that it can be referred in SnapshotDiffReport JSON schema.

var diffReportEntries = { "type": "object", "properties": { "sourcePath": { "description" : "Source path name relative to snapshot root", "type" : "string", "required" : true }, "targetPath": { "description" : "Target path relative to snapshot root used for renames", "type" : "string", "required" : true }, "type": { "description" : "Type of diff report entry", "enum" : ["CREATE", "MODIFY", "DELETE", "RENAME"], "required" : true } } }

SnapshotDiffReportListing JSON Schema

{ "name": "SnapshotDiffReportListing", "type": "object", "properties": { "SnapshotDiffReportListing": { "type" : "object", "properties" : { "isFromEarlier": { "description" : "the diff is calculated from older to newer snapshot or not", "type" : "boolean", "required" : true }, "lastIndex": { "description" : "the last index of listing iteration", "type" : "integer", "required" : true }, "lastPath": { "description" : "String representation of the last path of the listing iteration", "type" : "string", "required" : true }, "modifyList": { "description": "An array of DiffReportListingEntry", "type" : "array", "items" : diffReportListingEntries, "required" : true }, "createList": { "description": "An array of DiffReportListingEntry", "type" : "array", "items" : diffReportListingEntries, "required" : true }, "deleteList": { "description": "An array of DiffReportListingEntry", "type" : "array", "items" : diffReportListingEntries, "required" : true } } } } }

DiffReportListing Entries

JavaScript syntax is used to define diffReportEntries so that it can be referred in SnapshotDiffReport JSON schema.

var diffReportListingEntries = { "type": "object", "properties": { "dirId": { "description" : "inode id of the directory", "type" : "integer", "required" : true }, "fileId": { "description" : "inode id of the file", "type" : "integer", "required" : true }, "isRereference": { "description" : "this is reference or not", "type" : "boolean", "required" : true }, "sourcePath": { "description" : "string representation of path where changes have happened", "type" : "string", "required" : true }, "targetPath": { "description" : "string representation of target path of rename op", "type" : "string", "required" : false } } }

SnapshottableDirectoryList JSON Schema

{ "name": "SnapshottableDirectoryList", "type": "object", "properties": { "SnapshottableDirectoryList": { "description": "An array of SnapshottableDirectoryStatus", "type" : "array", "items" : snapshottableDirectoryStatus, "required" : true } } }

SnapshottableDirectoryStatus

JavaScript syntax is used to define snapshottableDirectoryStatus so that it can be referred in SnapshottableDirectoryList JSON schema.

var snapshottableDirectoryStatus = { "type": "object", "properties": { "dirStatus": fileStatusProperties, "parentFullPath": { "description" : "Full path of the parent of snapshottable directory", "type" : "string", "required" : true }, "snapshotNumber": { "description" : "Number of snapshots created on the snapshottable directory", "type" : "integer", "required" : true }, "snapshotQuota": { "description" : "Total number of snapshots allowed on the snapshottable directory", "type" : "integer", "required" : true } } }

SnapshotList JSON Schema

{ "name": "SnapshotList", "type": "object", "properties": { "SnapshotList": { "description": "An array of SnapshotStatus", "type" : "array", "items" : snapshotStatus, "required" : true } } }

SnapshotStatus

JavaScript syntax is used to define snapshotStatus so that it can be referred in SnapshotList JSON schema.

var snapshotStatus = { "type": "object", "properties": { "dirStatus": fileStatusProperties, "fullPath": { "description" : "Full path of the parent of the snapshot", "type" : "string", "required" : true }, "snapshotID": { "description" : "snapshot ID for the snapshot", "type" : "integer", "required" : true }, "deletionStatus": { "description" : "Status showing whether the snapshot is active or in deleted state", "type" : "string", "required" : true } } }

BlockLocations JSON Schema

A BlockLocations JSON object represents an array of BlockLocation JSON objects.

{ "name" : "BlockLocations", "properties": { "BlockLocations": { "type" : "object", "properties": { "BlockLocation": { "description": "An array of BlockLocation", "type" : "array", "items" : blockLocationProperties //See BlockLocation Properties } } } } }

See also BlockLocation Properties, GETFILEBLOCKLOCATIONS, BlockLocation

BlockLocation JSON Schema

{ "name" : "BlockLocation", "properties": { "BlockLocation": blockLocationProperties //See BlockLocation Properties } }

See also BlockLocation Properties, GETFILEBLOCKLOCATIONS, BlockLocation

BlockLocation Properties

JavaScript syntax is used to define blockLocationProperties so that it can be referred in both BlockLocation and BlockLocations JSON schemas.

var blockLocationProperties = { "type" : "object", "properties": { "cachedHosts": { "description": "Datanode hostnames with a cached replica", "type" : "array", "required" : "true", "items" : { "description": "A datanode hostname", "type" : "string" } }, "corrupt": { "description": "True if the block is corrupted", "type" : "boolean", "required" : "true" }, "hosts": { "description": "Datanode hostnames store the block", "type" : "array", "required" : "true", "items" : { "description": "A datanode hostname", "type" : "string" } }, "length": { "description": "Length of the block", "type" : "integer", "required" : "true" }, "names": { "description": "Datanode IP:xferPort for accessing the block", "type" : "array", "required" : "true", "items" : { "description": "DatanodeIP:xferPort", "type" : "string" } }, "offset": { "description": "Offset of the block in the file", "type" : "integer", "required" : "true" }, "storageTypes": { "description": "Storage type of each replica", "type" : "array", "required" : "true", "items" : { "description": "Storage type", "enum" : ["RAM_DISK", "SSD", "DISK", "ARCHIVE"] } }, "topologyPaths": { "description": "Datanode addresses in network topology", "type" : "array", "required" : "true", "items" : { "description": "/rack/host:ip", "type" : "string" } } } };

Server Defaults JSON Schema

{ "FsServerDefaults": { "replication": 3, "encryptDataTransfer": false, "defaultStoragePolicyId": 7, "writePacketSize": 65536, "fileBufferSize": 4096, "checksumType": 2, "trashInterval": 10080, "keyProviderUri": "", "blockSize": 134217728, "bytesPerChecksum": 512 } }

FsStatus JSON Schema

{ "FsStatus": { "used": 29229154304, "remaining": 292893392896, "capacity": 322122547200 } }

EC Policies JSON Schema

{ "ErasureCodingPolicies": { "ErasureCodingPolicyInfo": [ { "state": "ENABLED", "policy": { "name": "RS-6-3-1024k", "schema": { "codecName": "rs", "numDataUnits": 6, "numParityUnits": 3, "extraOptions": {} }, "cellSize": 1048576, "id": 1, "replicationPolicy": false, "codecName": "rs", "numDataUnits": 6, "numParityUnits": 3, "systemPolicy": true } } ] } }

EC Codecs JSON Schema

{ "ErasureCodingCodecs": { "rs": "rs_native, rs_java", "rs-legacy": "rs-legacy_java", "xor": "xor_native, xor_java" } }

Paths JSON Schema

{ "Paths": [{ "blocksize": 0, "owner": "hadoop", "path": "/user/user0/.Trash", "length": 0, "permission": "755", "modification_time": 1693050205747, "isdir": true, "block_replication": 0, "access_time": 0, "group": "supergroup" }] }

HTTP Query Parameter Dictionary

ACL Spec

Name aclspec
Description The ACL spec included in ACL modification operations.
Type String
Default Value
Valid Values See Permissions and HDFS.
Syntax See Permissions and HDFS.

XAttr Name

Name xattr.name
Description The XAttr name of a file/directory.
Type String
Default Value
Valid Values Any string prefixed with user./trusted./system./security..
Syntax Any string prefixed with user./trusted./system./security..

XAttr Value

Name xattr.value
Description The XAttr value of a file/directory.
Type String
Default Value
Valid Values An encoded value.
Syntax Enclosed in double quotes or prefixed with 0x or 0s.

See also: Extended Attributes

XAttr set flag

Name flag
Description The XAttr set flag.
Type String
Default Value
Valid Values CREATE,REPLACE.
Syntax CREATE,REPLACE.

See also: Extended Attributes

XAttr value encoding

Name encoding
Description The XAttr value encoding.
Type String
Default Value
Valid Values text hex base64
Syntax text hex base64

See also: Extended Attributes

Access Time

Name accesstime
Description The access time of a file/directory.
Type long
Default Value -1 (means keeping it unchanged)
Valid Values -1 or a timestamp
Syntax Any integer.

See also: SETTIMES

Block Size

Name blocksize
Description The block size of a file.
Type long
Default Value Specified in the configuration.
Valid Values > 0
Syntax Any integer.

See also: CREATE

Buffer Size

Name buffersize
Description The size of the buffer used in transferring data.
Type int
Default Value Specified in the configuration.
Valid Values > 0
Syntax Any integer.

See also: CREATE, APPEND, OPEN

Create Flag

Name createflag
Description Enum of possible flags to process while creating a file
Type enumerated strings
Default Value
Valid Values Legal combinations of create, overwrite, append and sync_block
Syntax See note below

The following combinations are not valid: * append,create * create,append,overwrite

See also: CREATE

Create Parent

Name createparent
Description If the parent directories do not exist, should they be created?
Type boolean
Default Value true
Valid Values true, false
Syntax true

See also: CREATESYMLINK

Delegation

Name delegation
Description The delegation token used for authentication.
Type String
Default Value
Valid Values An encoded token.
Syntax See the note below.

Note that delegation tokens are encoded as a URL safe string; see encodeToUrlString() and decodeFromUrlString(String) in org.apache.hadoop.security.token.Token for the details of the encoding.

See also: Authentication

Destination

Name destination
Description The destination path.
Type Path
Default Value (an invalid path)
Valid Values An absolute FileSystem path without scheme and authority.
Syntax Any path.

See also: CREATESYMLINK, RENAME

Do As

Name doas
Description Allowing a proxy user to do as another user.
Type String
Default Value null
Valid Values Any valid username.
Syntax Any string.

See also: Proxy Users

Fs Action

Name fsaction
Description File system operation read/write/execute
Type String
Default Value null (an invalid value)
Valid Values Strings matching regex pattern "[r-][w-][x-] "
Syntax "[r-][w-][x-] "

See also: CHECKACCESS,

Group

Name group
Description The name of a group.
Type String
Default Value (means keeping it unchanged)
Valid Values Any valid group name.
Syntax Any string.

See also: SETOWNER

Length

Name length
Description The number of bytes to be processed.
Type long
Default Value null (means the entire file)
Valid Values >= 0 or null
Syntax Any integer.

See also: OPEN

Modification Time

Name modificationtime
Description The modification time of a file/directory.
Type long
Default Value -1 (means keeping it unchanged)
Valid Values -1 or a timestamp
Syntax Any integer.

See also: SETTIMES

New Length

Name newlength
Description The size the file is to be truncated to.
Type long
Valid Values >= 0
Syntax Any long.

Offset

Name offset
Description The starting byte position.
Type long
Default Value 0
Valid Values >= 0
Syntax Any integer.

See also: OPEN

Old Snapshot Name

Name oldsnapshotname
Description The old name of the snapshot to be renamed.
Type String
Default Value null
Valid Values An existing snapshot name.
Syntax Any string.

See also: RENAMESNAPSHOT

Op

Name op
Description The name of the operation to be executed.
Type enum
Default Value null (an invalid value)
Valid Values Any valid operation name.
Syntax Any string.

See also: Operations

Overwrite

Name overwrite
Description If a file already exists, should it be overwritten?
Type boolean
Default Value false
Valid Values true
Syntax true

See also: CREATE

Owner

Name owner
Description The username who is the owner of a file/directory.
Type String
Default Value (means keeping it unchanged)
Valid Values Any valid username.
Syntax Any string.

See also: SETOWNER

Permission

Name permission
Description The permission of a file/directory.
Type Octal
Default Value 644 for files, 755 for directories
Valid Values 0 - 1777
Syntax Any radix-8 integer (leading zeros may be omitted.)

See also: CREATE, MKDIRS, SETPERMISSION

Recursive

Name recursive
Description Should the operation act on the content in the subdirectories?
Type boolean
Default Value false
Valid Values true
Syntax true

See also: RENAME

Renewer

Name renewer
Description The username of the renewer of a delegation token.
Type String
Default Value (means the current user)
Valid Values Any valid username.
Syntax Any string.

See also: GETDELEGATIONTOKEN

Replication

Name replication
Description The number of replications of a file.
Type short
Default Value Specified in the configuration.
Valid Values > 0
Syntax Any integer.

See also: CREATE, SETREPLICATION

Snapshot Name

Name snapshotname
Description The name of the snapshot to be created/deleted. Or the new name for snapshot rename.
Type String
Default Value null
Valid Values Any valid snapshot name.
Syntax Any string.

See also: CREATESNAPSHOT, DELETESNAPSHOT, RENAMESNAPSHOT

Sources

Name sources
Description A list of source paths.
Type String
Default Value
Valid Values A list of comma separated absolute FileSystem paths without scheme and authority.
Syntax Any string.

See also: CONCAT

Token

Name token
Description The delegation token used for the operation.
Type String
Default Value
Valid Values An encoded token.
Syntax See the note in Delegation.

See also: RENEWDELEGATIONTOKEN, CANCELDELEGATIONTOKEN

Token Kind

Name kind
Description The kind of the delegation token requested
Type String
Default Value (Server sets the default kind for the service)
Valid Values A string that represents token kind e.g “HDFS_DELEGATION_TOKEN” or “WEBHDFS delegation”
Syntax Any string.

See also: GETDELEGATIONTOKEN

Token Service

Name service
Description The name of the service where the token is supposed to be used, e.g. ip:port of the namenode
Type String
Default Value
Valid Values ip:port in string format or logical name of the service
Syntax Any string.

See also: GETDELEGATIONTOKEN

Username

Name user.name
Description The authenticated user; see Authentication.
Type String
Default Value null
Valid Values Any valid username.
Syntax Any string.

See also: Authentication

NoRedirect

Name noredirect
Description Whether the response should return an HTTP 307 redirect or HTTP 200 OK. See Create and Write to a File.
Type boolean
Default Value false
Valid Values true
Syntax true

See also: Create and Write to a File

Namespace Quota

Name namespacequota
Description Limit on the namespace usage, i.e., number of files/directories, under a directory.
Type String
Default Value Long.MAX_VALUE
Valid Values > 0.
Syntax Any integer.

See also: SETQUOTA

Storage Space Quota

Name storagespacequota
Description Limit on storage space usage (in bytes, including replication) under a directory.
Type String
Default Value Long.MAX_VALUE
Valid Values > 0.
Syntax Any integer.

See also: SETQUOTA, SETQUOTABYSTORAGETYPE

Storage Type

Name storagetype
Description Storage type of the specific storage type quota to be modified.
Type String
Default Value
Valid Values Any valid storage type.
Syntax Any string.

See also: SETQUOTABYSTORAGETYPE

Storage Policy

Name storagepolicy
Description The name of the storage policy.
Type String
Default Value
Valid Values Any valid storage policy name; see GETALLSTORAGEPOLICY.
Syntax Any string.

See also: SETSTORAGEPOLICY

Erasure Coding Policy

Name ecpolicy
Description The name of the erasure coding policy.
Type String
Default Value
Valid Values Any valid erasure coding policy name;
Syntax Any string.

See also: ENABLEECPOLICY or DISABLEECPOLICY

Start After

Name startAfter
Description The last item returned in the liststatus batch.
Type String
Default Value
Valid Values Any valid file/directory name.
Syntax Any string.

See also: LISTSTATUS_BATCH