CLI - PyIceberg (original) (raw)

Python CLI¶

Pyiceberg comes with a CLI that's available after installing the pyiceberg package.

You can pass the path to the Catalog using the --uri and --credential argument, but it is recommended to setup a ~/.pyiceberg.yaml config as described in the Catalog section.

[](#%5F%5Fcodelineno-0-1)➜ pyiceberg --help [](#%5F%5Fcodelineno-0-2)Usage: pyiceberg [OPTIONS] COMMAND [ARGS]... [](#%5F%5Fcodelineno-0-3) [](#%5F%5Fcodelineno-0-4)Options: [](#%5F%5Fcodelineno-0-5) --catalog TEXT [](#%5F%5Fcodelineno-0-6) --verbose BOOLEAN [](#%5F%5Fcodelineno-0-7) --output [text|json] [](#%5F%5Fcodelineno-0-8) --ugi TEXT [](#%5F%5Fcodelineno-0-9) --uri TEXT [](#%5F%5Fcodelineno-0-10) --credential TEXT [](#%5F%5Fcodelineno-0-11) --help Show this message and exit. [](#%5F%5Fcodelineno-0-12) [](#%5F%5Fcodelineno-0-13)Commands: [](#%5F%5Fcodelineno-0-14) create Operation to create a namespace. [](#%5F%5Fcodelineno-0-15) describe Describe a namespace or a table. [](#%5F%5Fcodelineno-0-16) drop Operations to drop a namespace or table. [](#%5F%5Fcodelineno-0-17) files List all the files of the table. [](#%5F%5Fcodelineno-0-18) list List tables or namespaces. [](#%5F%5Fcodelineno-0-19) list-refs List all the refs in the provided table. [](#%5F%5Fcodelineno-0-20) location Return the location of the table. [](#%5F%5Fcodelineno-0-21) properties Properties on tables/namespaces. [](#%5F%5Fcodelineno-0-22) rename Rename a table. [](#%5F%5Fcodelineno-0-23) schema Get the schema of the table. [](#%5F%5Fcodelineno-0-24) spec Return the partition spec of the table. [](#%5F%5Fcodelineno-0-25) uuid Return the UUID of the table. [](#%5F%5Fcodelineno-0-26) version Print pyiceberg version.

This example assumes that you have a default catalog set. If you want to load another catalog, for example, the rest example above. Then you need to set --catalog rest.

[](#%5F%5Fcodelineno-1-1)➜ pyiceberg list [](#%5F%5Fcodelineno-1-2)default [](#%5F%5Fcodelineno-1-3)nyc

[](#%5F%5Fcodelineno-2-1)➜ pyiceberg list nyc [](#%5F%5Fcodelineno-2-2)nyc.taxis

[](#%5F%5Fcodelineno-3-1)➜ pyiceberg describe nyc.taxis [](#%5F%5Fcodelineno-3-2)Table format version 1 [](#%5F%5Fcodelineno-3-3)Metadata location file:/.../nyc.db/taxis/metadata/00000-aa3a3eac-ea08-4255-b890-383a64a94e42.metadata.json [](#%5F%5Fcodelineno-3-4)Table UUID 6cdfda33-bfa3-48a7-a09e-7abb462e3460 [](#%5F%5Fcodelineno-3-5)Last Updated 1661783158061 [](#%5F%5Fcodelineno-3-6)Partition spec [] [](#%5F%5Fcodelineno-3-7)Sort order [] [](#%5F%5Fcodelineno-3-8)Current schema Schema, id=0 [](#%5F%5Fcodelineno-3-9)├── 1: VendorID: optional long [](#%5F%5Fcodelineno-3-10)├── 2: tpep_pickup_datetime: optional timestamptz [](#%5F%5Fcodelineno-3-11)├── 3: tpep_dropoff_datetime: optional timestamptz [](#%5F%5Fcodelineno-3-12)├── 4: passenger_count: optional double [](#%5F%5Fcodelineno-3-13)├── 5: trip_distance: optional double [](#%5F%5Fcodelineno-3-14)├── 6: RatecodeID: optional double [](#%5F%5Fcodelineno-3-15)├── 7: store_and_fwd_flag: optional string [](#%5F%5Fcodelineno-3-16)├── 8: PULocationID: optional long [](#%5F%5Fcodelineno-3-17)├── 9: DOLocationID: optional long [](#%5F%5Fcodelineno-3-18)├── 10: payment_type: optional long [](#%5F%5Fcodelineno-3-19)├── 11: fare_amount: optional double [](#%5F%5Fcodelineno-3-20)├── 12: extra: optional double [](#%5F%5Fcodelineno-3-21)├── 13: mta_tax: optional double [](#%5F%5Fcodelineno-3-22)├── 14: tip_amount: optional double [](#%5F%5Fcodelineno-3-23)├── 15: tolls_amount: optional double [](#%5F%5Fcodelineno-3-24)├── 16: improvement_surcharge: optional double [](#%5F%5Fcodelineno-3-25)├── 17: total_amount: optional double [](#%5F%5Fcodelineno-3-26)├── 18: congestion_surcharge: optional double [](#%5F%5Fcodelineno-3-27)└── 19: airport_fee: optional double [](#%5F%5Fcodelineno-3-28)Current snapshot Operation.APPEND: id=5937117119577207079, schema_id=0 [](#%5F%5Fcodelineno-3-29)Snapshots Snapshots [](#%5F%5Fcodelineno-3-30)└── Snapshot 5937117119577207079, schema 0: file:/.../nyc.db/taxis/metadata/snap-5937117119577207079-1-94656c4f-4c66-4600-a4ca-f30377300527.avro [](#%5F%5Fcodelineno-3-31)Properties owner root [](#%5F%5Fcodelineno-3-32)write.format.default parquet

Or output in JSON for automation:

[](#%5F%5Fcodelineno-4-1)➜ pyiceberg --output json describe nyc.taxis | jq [](#%5F%5Fcodelineno-4-2){ [](#%5F%5Fcodelineno-4-3) "identifier": [ [](#%5F%5Fcodelineno-4-4) "nyc", [](#%5F%5Fcodelineno-4-5) "taxis" [](#%5F%5Fcodelineno-4-6) ], [](#%5F%5Fcodelineno-4-7) "metadata_location": "file:/.../nyc.db/taxis/metadata/00000-aa3a3eac-ea08-4255-b890-383a64a94e42.metadata.json", [](#%5F%5Fcodelineno-4-8) "metadata": { [](#%5F%5Fcodelineno-4-9) "location": "file:/.../nyc.db/taxis", [](#%5F%5Fcodelineno-4-10) "table-uuid": "6cdfda33-bfa3-48a7-a09e-7abb462e3460", [](#%5F%5Fcodelineno-4-11) "last-updated-ms": 1661783158061, [](#%5F%5Fcodelineno-4-12) "last-column-id": 19, [](#%5F%5Fcodelineno-4-13) "schemas": [ [](#%5F%5Fcodelineno-4-14) { [](#%5F%5Fcodelineno-4-15) "type": "struct", [](#%5F%5Fcodelineno-4-16) "fields": [ [](#%5F%5Fcodelineno-4-17) { [](#%5F%5Fcodelineno-4-18) "id": 1, [](#%5F%5Fcodelineno-4-19) "name": "VendorID", [](#%5F%5Fcodelineno-4-20) "type": "long", [](#%5F%5Fcodelineno-4-21) "required": false [](#%5F%5Fcodelineno-4-22) }, [](#%5F%5Fcodelineno-4-23)... [](#%5F%5Fcodelineno-4-24) { [](#%5F%5Fcodelineno-4-25) "id": 19, [](#%5F%5Fcodelineno-4-26) "name": "airport_fee", [](#%5F%5Fcodelineno-4-27) "type": "double", [](#%5F%5Fcodelineno-4-28) "required": false [](#%5F%5Fcodelineno-4-29) } [](#%5F%5Fcodelineno-4-30) ], [](#%5F%5Fcodelineno-4-31) "schema-id": 0, [](#%5F%5Fcodelineno-4-32) "identifier-field-ids": [] [](#%5F%5Fcodelineno-4-33) } [](#%5F%5Fcodelineno-4-34) ], [](#%5F%5Fcodelineno-4-35) "current-schema-id": 0, [](#%5F%5Fcodelineno-4-36) "partition-specs": [ [](#%5F%5Fcodelineno-4-37) { [](#%5F%5Fcodelineno-4-38) "spec-id": 0, [](#%5F%5Fcodelineno-4-39) "fields": [] [](#%5F%5Fcodelineno-4-40) } [](#%5F%5Fcodelineno-4-41) ], [](#%5F%5Fcodelineno-4-42) "default-spec-id": 0, [](#%5F%5Fcodelineno-4-43) "last-partition-id": 999, [](#%5F%5Fcodelineno-4-44) "properties": { [](#%5F%5Fcodelineno-4-45) "owner": "root", [](#%5F%5Fcodelineno-4-46) "write.format.default": "parquet" [](#%5F%5Fcodelineno-4-47) }, [](#%5F%5Fcodelineno-4-48) "current-snapshot-id": 5937117119577207000, [](#%5F%5Fcodelineno-4-49) "snapshots": [ [](#%5F%5Fcodelineno-4-50) { [](#%5F%5Fcodelineno-4-51) "snapshot-id": 5937117119577207000, [](#%5F%5Fcodelineno-4-52) "timestamp-ms": 1661783158061, [](#%5F%5Fcodelineno-4-53) "manifest-list": "file:/.../nyc.db/taxis/metadata/snap-5937117119577207079-1-94656c4f-4c66-4600-a4ca-f30377300527.avro", [](#%5F%5Fcodelineno-4-54) "summary": { [](#%5F%5Fcodelineno-4-55) "operation": "append", [](#%5F%5Fcodelineno-4-56) "spark.app.id": "local-1661783139151", [](#%5F%5Fcodelineno-4-57) "added-data-files": "1", [](#%5F%5Fcodelineno-4-58) "added-records": "2979431", [](#%5F%5Fcodelineno-4-59) "added-files-size": "46600777", [](#%5F%5Fcodelineno-4-60) "changed-partition-count": "1", [](#%5F%5Fcodelineno-4-61) "total-records": "2979431", [](#%5F%5Fcodelineno-4-62) "total-files-size": "46600777", [](#%5F%5Fcodelineno-4-63) "total-data-files": "1", [](#%5F%5Fcodelineno-4-64) "total-delete-files": "0", [](#%5F%5Fcodelineno-4-65) "total-position-deletes": "0", [](#%5F%5Fcodelineno-4-66) "total-equality-deletes": "0" [](#%5F%5Fcodelineno-4-67) }, [](#%5F%5Fcodelineno-4-68) "schema-id": 0 [](#%5F%5Fcodelineno-4-69) } [](#%5F%5Fcodelineno-4-70) ], [](#%5F%5Fcodelineno-4-71) "snapshot-log": [ [](#%5F%5Fcodelineno-4-72) { [](#%5F%5Fcodelineno-4-73) "snapshot-id": "5937117119577207079", [](#%5F%5Fcodelineno-4-74) "timestamp-ms": 1661783158061 [](#%5F%5Fcodelineno-4-75) } [](#%5F%5Fcodelineno-4-76) ], [](#%5F%5Fcodelineno-4-77) "metadata-log": [], [](#%5F%5Fcodelineno-4-78) "sort-orders": [ [](#%5F%5Fcodelineno-4-79) { [](#%5F%5Fcodelineno-4-80) "order-id": 0, [](#%5F%5Fcodelineno-4-81) "fields": [] [](#%5F%5Fcodelineno-4-82) } [](#%5F%5Fcodelineno-4-83) ], [](#%5F%5Fcodelineno-4-84) "default-sort-order-id": 0, [](#%5F%5Fcodelineno-4-85) "refs": { [](#%5F%5Fcodelineno-4-86) "main": { [](#%5F%5Fcodelineno-4-87) "snapshot-id": 5937117119577207000, [](#%5F%5Fcodelineno-4-88) "type": "branch" [](#%5F%5Fcodelineno-4-89) } [](#%5F%5Fcodelineno-4-90) }, [](#%5F%5Fcodelineno-4-91) "format-version": 1, [](#%5F%5Fcodelineno-4-92) "schema": { [](#%5F%5Fcodelineno-4-93) "type": "struct", [](#%5F%5Fcodelineno-4-94) "fields": [ [](#%5F%5Fcodelineno-4-95) { [](#%5F%5Fcodelineno-4-96) "id": 1, [](#%5F%5Fcodelineno-4-97) "name": "VendorID", [](#%5F%5Fcodelineno-4-98) "type": "long", [](#%5F%5Fcodelineno-4-99) "required": false [](#%5F%5Fcodelineno-4-100) }, [](#%5F%5Fcodelineno-4-101)... [](#%5F%5Fcodelineno-4-102) { [](#%5F%5Fcodelineno-4-103) "id": 19, [](#%5F%5Fcodelineno-4-104) "name": "airport_fee", [](#%5F%5Fcodelineno-4-105) "type": "double", [](#%5F%5Fcodelineno-4-106) "required": false [](#%5F%5Fcodelineno-4-107) } [](#%5F%5Fcodelineno-4-108) ], [](#%5F%5Fcodelineno-4-109) "schema-id": 0, [](#%5F%5Fcodelineno-4-110) "identifier-field-ids": [] [](#%5F%5Fcodelineno-4-111) }, [](#%5F%5Fcodelineno-4-112) "partition-spec": [] [](#%5F%5Fcodelineno-4-113) } [](#%5F%5Fcodelineno-4-114)}

You can also add, update or remove properties on tables or namespaces:

[](#%5F%5Fcodelineno-5-1)➜ pyiceberg properties set table nyc.taxis write.metadata.delete-after-commit.enabled true [](#%5F%5Fcodelineno-5-2)Set write.metadata.delete-after-commit.enabled=true on nyc.taxis [](#%5F%5Fcodelineno-5-3) [](#%5F%5Fcodelineno-5-4)➜ pyiceberg properties get table nyc.taxis [](#%5F%5Fcodelineno-5-5)write.metadata.delete-after-commit.enabled true [](#%5F%5Fcodelineno-5-6) [](#%5F%5Fcodelineno-5-7)➜ pyiceberg properties remove table nyc.taxis write.metadata.delete-after-commit.enabled [](#%5F%5Fcodelineno-5-8)Property write.metadata.delete-after-commit.enabled removed from nyc.taxis [](#%5F%5Fcodelineno-5-9) [](#%5F%5Fcodelineno-5-10)➜ pyiceberg properties get table nyc.taxis write.metadata.delete-after-commit.enabled [](#%5F%5Fcodelineno-5-11)Could not find property write.metadata.delete-after-commit.enabled on nyc.taxis