Skip to content

Normalise a JSON column into typed columns

normalise_json infers a schema for a JSON string column, rewrites every row to fit it, and decodes the result into typed Polars columns.

Typed columns (the default)

import polars as pl
import polars_genson

df = pl.DataFrame({"json": [
    '{"id": 1, "name": "Ada", "tags": ["x"]}',
    '{"id": 2, "name": "Bo", "email": "bo@example.com"}',
]})
print(df.genson.normalise_json("json"))

shape: (2, 4)
┌─────┬──────┬───────────┬────────────────┐
│ id  ┆ name ┆ tags      ┆ email          │
│ --- ┆ ---  ┆ ---       ┆ ---            │
│ i64 ┆ str  ┆ list[str] ┆ str            │
╞═════╪══════╪═══════════╪════════════════╡
│ 1   ┆ Ada  ┆ ["x"]     ┆ null           │
│ 2   ┆ Bo   ┆ null      ┆ bo@example.com │
└─────┴──────┴───────────┴────────────────┘

Each field becomes a column, and rows that lack a field get null.

One struct column: unnest=False

To keep the result as a single struct column, pass unnest=False. The result is a one-column DataFrame, so it can sit alongside the original columns:

print(df.hstack(df.genson.normalise_json("json", unnest=False).rename({"json": "parsed"})))

shape: (2, 2)
┌─────────────────────────────────┬────────────────────────────────┐
│ json                            ┆ parsed                         │
│ ---                             ┆ ---                            │
│ str                             ┆ struct[4]                      │
╞═════════════════════════════════╪════════════════════════════════╡
│ {"id": 1, "name": "Ada", "tags… ┆ {1,"Ada",["x"],null}           │
│ {"id": 2, "name": "Bo", "email… ┆ {2,"Bo",null,"bo@example.com"} │
└─────────────────────────────────┴────────────────────────────────┘

JSON strings: decode=False

decode=False returns the normalised rows as JSON strings. Every row then has every field:

print(df.genson.normalise_json("json", decode=False).to_list())

['{"id":1,"name":"Ada","tags":["x"],"email":null}', '{"id":2,"name":"Bo","tags":null,"email":"bo@example.com"}']

A schema you already have: decode=schema

If you already know the schema, for example from infer_polars_schema on an earlier batch, pass it as decode. genson decodes with it directly and skips inferring the schema for decoding:

schema = df.genson.infer_polars_schema("json")
print(df.genson.normalise_json("json", decode=schema))

shape: (2, 4)
┌─────┬──────┬───────────┬────────────────┐
│ id  ┆ name ┆ tags      ┆ email          │
│ --- ┆ ---  ┆ ---       ┆ ---            │
│ i64 ┆ str  ┆ list[str] ┆ str            │
╞═════╪══════╪═══════════╪════════════════╡
│ 1   ┆ Ada  ┆ ["x"]     ┆ null           │
│ 2   ┆ Bo   ┆ null      ┆ bo@example.com │
└─────┴──────┴───────────┴────────────────┘

Options that change the output

  • empty_as_null (default True): empty arrays and maps become null. See Nulls and empty values.
  • coerce_strings: parse numbers and booleans written as strings. See Mixed types.
  • map_threshold, force_field_types, unify_maps, map_encoding: how objects become maps or records. See Maps and records.
  • wrap_scalars and force_scalar_promotion: how scalars that collide with objects are kept. See Mixed types.

For data in Parquet files, normalise_from_parquet does the same without loading the data into a DataFrame.